Troubleshooting upload issues caused by special files

Owned by Andy Tseng
Mar 21, 2024
2 min read

Sparse files

Sparse Files

Sparse files are files that have large sections of unallocated data. They are commonly used in Linux/Unix systems. Sparse files use storage efficiently when the files have a lot of holes (contiguous ranges of bytes having the value of zero) by storing only metadata for the holes instead of using real disk blocks.

Sparse files should be either excluded, or compressed before uploading to Mediaflux. As Mediaflux backend does not support sparse files and treats them as regular files. Uploading uncompressed sparse files will be waste of storage space. We've seen issues caused by very large sparse files. 

Find sparse files

To find the sparse files in your file system, you can use find  command below:

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print $2}'

Compress sparse files

If you are aware of sparse files in your local file system, you can run the following command to compress them before uploading to Mediaflux:

find ./ -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print $2}' | xargs -I {} sh -c "tar -Sczvf {}.tar.gz {}; rm -f {}"

Warning

The above command compresses the sparse files to *.tar.gz files and preserve their holes (-S option for tar), and the original sparse files will be replaced. 

DO NOT try it if you don't know what you are doing.


FIFO (Named Pipe)

A FIFO (First In First Out) is similar to a pipe. The principal difference is that a FIFO has a name within the file system and is opened in the same way as a regular file. A FIFO has a write end and a read end, and data is read from the pipe in the same order as it is written. Fifo is also termed as Named pipes in Linux.

FIFO should not be uploaded to Mediaflux.


Mediaflux Explorer

Uploading FIFO causes Mediaflux Explorer (current version: v1.5.6) to crash.


unimelb-mf-upload (in unimelb-mf-clients)

Early versions (prior to v0.7.4) of unimelb-mf-upload also hangs when uploading FIFO. 

From version v0.7.4 and above, unimelb-mf-upload excludes FIFO files.

Find FIFO (Named Pipes)

The following command can be used to list the FIFO files in your file system:

find ./ -type p