New simplified config file format Look at gocfl2.toml
go install github.com/ocfl-archive/gocfl/v2/gocfl@latest
This library supports the Oxford Common Filesystem Layout (OCFL) and focuses on creation, update, validation and extraction of ocfl StorageRoots and Objects.
GOCFL command line tool supports the following subcommands
There's a quickstart guide available.
There are several OCFL tools & libraries which already exists. This software is build with the following motivation.
Regarding performance, Storage I/O generates the main performance issues. Therefor, every file should be read and written only once. Only in case of deduplication, the checksum of a file is calculated before ingest and a second time while ingesting.
Serialization of an OCFL Storage Root into a container format like ZIP must not generate overhead on disk I/O. Therefor generation of an OCFL Container is possible without an intermediary OCFL Storage Root on a filesystem.
For storing OCFL containers in low-security locations (cloud storage, etc.), it's possible to create an AES-256 encrypted container on ingest.
The extensions described in the OCFL standard are quite open in their functionality and may belong to the Storage Root or Object. Since there's no specification of a generic extension api, it's difficult to integrate specific extension hooks into other libraries. This library identifies 7 different extension hooks so far.
When content is ingested into OCFL objects, technical metadata should be extracted and stored alongside the manifest data. This allows technical metadata to be extracted alongside the content. Since the OCFL structure is quite rigid, there's a need for a special extension to support this.
- Supports local filesystems
- Supports S3 Cloud Storage (via MinIO Client SDK)
- SFTP Storage
- Google Cloud Storage
- Serialization into ZIP Container
- AES Encryption of Container
- Supports mixing of source and target storage systems
- Non blocking validation (does not stop on validation errors)
- Support for OCFL v1.0 and v1.1
- Documentation for API
- Digest Algorithms for Manifest: SHA512, SHA256
- Fixity Algorithms: SHA1, SHA256, SHA512, BLAKE2b-160, BLAKE2b-256, BLAKE2b-384, BLAKE2b-512, MD5
- Concurrent checksum generation on ingest/extract (multi-threaded)
- Minimized I/O (data is read and written only once on Object creation)
- Update strategy echo (incl. deletions) and contribute
- Deduplication (needs double read of all content files, switchable)
- Nearly full coverage of validation errors and warnings
- Content information
- Extraction with version selection
- Display of content via Webserver
- Report generation
- Community Extensions
- 0001-digest-algorithms
- 0002-flat-direct-storage-layout
- 0003-hash-and-id-n-tuple-storage-layout
- 0004-hashed-n-tuple-storage-layout
- 0005-mutable-head
- 0006-flat-omit-prefix-storage-layout
- 0007-n-tuple-omit-prefix-storage-layout
- 0008-schema-registry
- Local Extensions
- NNNN-pairtree-storage-layout
- NNNN-direct-clean-path-layout
- NNNN-content-subpath (integration of non-payload files in content)
- NNNN-metafile (integration of a metadata file)
- NNNN-mets (generation of mets and premis files)
- NNNN-indexer (technical metadata indexing)
- NNNN-migration (migration of file formats)
- NNNN-gocfl-extension-manager (initial extension for sorted exclusion and sorted execution)
- NNNN-filesystem (filesystem metadata extension)
- NNNN-thumbnail (generation of thumbnails)
A fast and reliable OCFL creator, extractor and validator.
https://github.com/ocfl-archive/gocfl
Jürgen Enge (University Library Basel, juergen@info-age.net)
Version v1.0-beta.7
Usage:
gocfl [flags]
gocfl [command]
Available Commands:
add adds new object to existing ocfl structure
completion Generate the autocompletion script for the specified shell
create creates a new ocfl structure with initial content of one object
display show content of ocfl object in webbrowser
extract extract version of ocfl content
extractmeta extract metadata from ocfl structure
help Help about any command
init initializes an empty ocfl structure
stat statistics of an ocfl structure
update update object in existing ocfl structure
validate validates an ocfl structure
Flags:
--config string config file (default is $HOME/.gocfl.toml)
-h, --help help for gocfl
--log-file string log output file (default is console)
--log-level string log level (CRITICAL|ERROR|WARNING|NOTICE|INFO|DEBUG) (default "ERROR")
--s3-access-key-id string Access Key ID for S3 Buckets
--s3-endpoint string Endpoint for S3 Buckets
--s3-region string Region for S3 Access
--s3-secret-access-key string Secret Access Key for S3 Buckets
--with-indexer starts indexer as a local service
Use "gocfl [command] --help" for more information about a command.