Binary repository manager

A binary repository manager is a software tool designed to optimize the download and storage of binary files used and produced in software development. It centralizes the management of all the binary artifacts generated and used by the organization to overcome the complexity arising from the diversity of binary artifact types, their position in the overall workflow and the dependencies between them.

A binary repository is a software repository for packages, artifacts and their corresponding metadata. It can be used to store binary files produced by an organization itself, such as product releases and nightly product builds, or for third party binaries which must be treated differently for both technical and legal reasons.

Introduction

Software development can be a complex process^[1][2] involving many developers, or teams of developers working on shared code bases, accessing the same build tools, downloading and using a shared set of binary resources, and deploying components into the same software product. To manage the source files used in software development, organizations will typically use revision control. The many source files used in software development are eventually built into the binary artifacts (also known as “binaries”) which constitute the components of a software product. In addition, in order to provide their functionality and feature set, software products may use many 3rd party artifacts downloaded from free open source repositories or purchased from commercial sources.^[3] Consequently, a software product may comprise tens, hundreds and even thousands of individual binary artifacts which must be managed in order to efficiently maintain a coherent and functional software product. This function of managing the binary artifacts is done by a binary repository manager. A binary repository manager can be thought of as being to binaries what revision control is to source files.

Universal package manager

The software and technology industry continues to change and grow, binary repository managers are no different. They are beginning to shift towards positioning as a universal package managers.^[4] These package managers aim to standardize the way enterprises treat all package types used in the software development process. They give users the ability to apply security and compliance metrics across all artifact types. Universal package managers have been referred to as being at the center of a DevOps toolchain.^[5]

Notable Universal package managers include:^[6]

• Apache Archiva
• Jfrog's Artifactory
• Inedo's ProGet
• Sonatype Nexus

Relationship to continuous integration

As part of the development lifecycle, source code is continuously being built into binary artifacts using continuous integration. This may interact with a binary repository manager much like a developer would by getting artifacts from the repositories and pushing builds there. Tight integration with CI servers enables the storage of important metadata such as:

• Which user triggered the build (whether manually or by committing to revision control)
• Which modules were built
• Which sources were used (commit id, revision, branch)
• Dependencies used
• Environment variables
• Packages installed

Artifacts and packages

Artifacts and packages inherently mean different things. Artifacts are simply an output or collection of files (ex. JAR, WAR, DLLS, RPM etc.) and one of those files may contain metadata (e.g. POM file). Whereas packages are a single archive file in a well-defined format (ex. NuGet) that contain files appropriate for the package type (ex. DLL, PDB).^[7] Many artifacts result from builds but other types are crucial as well. Packages are essentially one of two things: a library or an application.^[8]

Compared to source files, binary artifacts are often larger by orders of magnitude, they are rarely deleted or overwritten (except for rare cases such as snapshots or nightly builds), and they are usually accompanied by lots of metadata such as id, package name, version, license and more.

Metadata

Metadata describes a binary artifact, is stored and specified separately from the artifact itself, and can have several additional uses. The following table shows some common metadata types and their uses:

Key features of repository managers

Key factors and features when considering the adoption of a package manager include:^[9]

• Caching – Caching simply stores local copies of packages. This increases performance for slow internet connections by allowing the user to pull from the local repository instead of externally. Caching locally allows frequently used packages to be available even during times of external repository outages.
• Retention policies - Repository managers can be used and configured to support organization purging policies to ensure reasonable disk space usage. Local repositories for third party artifacts may also be purged after not being used by any release for a specified time.
• License filtering - Third party artifacts may be subject to approval processes due to licensing and legal issues. Package managers allow for the restriction of only approved artifacts into deployment.
• High availability - Since a binary repository manager maintains all the development dependencies, it is vital to always maintain access to these artifacts. Any down-time of the binary repository manager can halt development with all the significant consequences to the organization. A high availability instance allows an enterprise to overcome the risk associated with downtime, through automatic failover. This is achieved by having a redundant set of repository managers work against the same database and file storage. Maintaining enterprise wide stability and performance at all times
• User restrictions - Repository managers can be integrated with other organizational systems such as LDAP or Single Sign-On servers to simplify and centralize user management. This gives an enterprise granular control over who has access to vital software components.

See also

• List of software package management systems

References