Research Support Handbook

How can you store research data effectively?

Collect & Store

Process & Analyse

Best practices for data storage.

This page aims to provide you with some practical technical and organisational guidelines for storing research data. In general, you should keep in mind that data should be easy to work with for both humans and computers.

Avoiding technical issues

To improve readability of your research data and avoid technical issues it is good to follow some best practices in naming your files and folders.

Note

This guide is meant as a guideline. If your source data does not adhere to these practices, it is not always feasible to rename everything. The VU provided storage platforms will be able to store the data properly, but be aware of the issues mentioned.

Only use lowercase characters

Linux and Yoda are case sensitive, but Windows and MacOS are not, which leads to unexpected behavior when transferring data. For this reason it is best to stick to lowercase characters in file and folder names, especially when creating data on a Linux system such as ADA or Snellius.

Avoid spaces

Use - or _ instead of spaces, this keeps file and folder names readable without sacrificing accessibility.

Accessing data from the CLI (Command Line Interface) or a script is a lot easier if you avoid spaces. Have a look at the following example. If you try to view the contents of the file run 1/lab output.txt with the cat command, you will get the following result:

~$ cat run 1/lab output.txt
cat: run: No such file or directory
cat: 1/lab: No such file or directory
cat: output.txt: No such file or directory

You must either “escape” the spaces or put quotes around the path to make the command work:

~$ cat run\ 1/lab\ output.txt
Output from lab equipment

~$ cat "run 1/lab output.txt"
Output from lab equipment

If you avoid spaces the path is just as clearly readable, but easier to access:

~$ cat "run1/lab_output.txt"
Output from lab equipment

Avoid other special characters

Modern filesystems can deal with special characters, such as ! # $ % & ' " ` @ ^ ~ + , . ; = ) (, in file and folder names, but applications might have trouble. Special characters, like spaces, also often need to be “escaped” on the command line.

Keep file and folder names as short as possible

File and foldernames should be short but descriptive. Avoid nesting subfolders too deep.

Windows in some cases has a maximum path (folder, subfolders and file name) length of 250 characters. You could create a much longer path on a Linux system (such as ADA or Snellius) and find you are unable to download it to a Windows laptop.

Some practical advice on file naming practices illustrated with examples (good and bad):

The repository by Jenny Bryan provides examples of good and bad file naming practices
The CESSDA Data Management Expert Guide discusses file name elements and best practices, and provides examples as well

Working with data on your laptop

SciStor allows you to “mount” (assign a driveletter to) your project folder. This is fast enough to be able to directly edit files without copying them to your local drive first.

Connections to cloud storage such as Research Drive or Yoda are not fast enough to directly edit files. This means you will primarily store and work with a copy of the data on your laptop. Create a dedicated folder for your research data.

Warning

Make sure to avoid storing data in a folder synced to OneDrive, such as the Documents folder on Windows, especially if you are working with some type of sensitive data (e.g. personal data, data to which knowledge security regulations apply, data under contractual agreements). OneDrive may not be secure enough for these types of data, so it shouldn’t end up there unintentionally.

For Research Drive you can configure the Nextcloud client so it automatically syncs your data folder to Research Drive. Make sure to regurarly check the web interface and client to ensure the sync process is still functioning.

For Yoda you will need to manually transfer data to your laptop. Create a dedicated folder to make this easier. The Research Support Handbook has a page with tips on ways to work with data stored on Yoda.

Make a plan

When working with collaborators, you should make a practical plan on how to work with your project’s data. But even if you are the only one working on the project it is useful to decide on a useful data structure and naming scheme.

Agree on a naming convention and folder structure

Some topics to decide on:

Hyphens or underscores? Both can be used to separate file name elements; it is useful to be consistent.
Dates in file and folder names? If using dates: YYYYMMDD is unambiguous, short and sorts well.
Version numbers in file names? Avoid using FINAL, it never is.
Prefixes for document types? INV for invoices, etc. This will make finding files easier.
A folder structure. Where you will put which data. The CESSDA Data Management Expert Guide provides some practical advice for folder structures. It is focussed on Social Sciences data, but its insights can be applied to other types of research as well. If the repository in which you would like to preserve your data requires a certain folder structure, you will save time if you apply this folder structure from the beginning.

Agree on how to work with files

Most tools and storage platforms do not allow you to work with others on files simultaneously. This means the last person who saves a file overwrites a colleague’s edits. In the best case a file will be “locked” so only one user at a time can access it.

Make sure you communicate with your collaborators who works on what files and when, choosing a good folder structure helps with that.

Document your choices

Document relevant choices in a README file so your collaborators know what to do and improve reusability of the dataset.