particle-libraries-v2.md

Particle Libraries v2

Abstract

Current user libraries architecture wasn't updated since its conception. Since then we added new tools (CLI and Dev), new boards and welcomed 3rd party module vendors in our ecosystem. All not anticipated when designing them. We want to revisit this with next version of our libraries.

Tools created for this purpose can be used outside our ecosystem i.e. with Arduino, Raspberry Pi.

Goals

• allow libraries to be available outside Build (CLI, Dev)
• make libraries Arduino compatible (both ways)
• fix
  • #95
  • need for re-adding library to update it
  • issues with libraries not compatible with anything other than Core
  • need for bundling other libraries in yours
  • #9

Proposed library structure

All starts with "everything is a library" mantra (similar to modules in npm). Every user project and library has to contain library.properties file in its root directory. Instead of raising error, tools should suggest generating a plain one.

library.properties

Example file can be found in uber-library-example/feature/libraries-v2. It's an evolution of spark.json file and is compatible with Arduino IDE 1.5: Library specification.

Library version string has to be semver compatible.

Storing structured data should be done using a dot notation in keys i.e.:

architectures=avr,particle-core,particle-photon
dependencies.neopixel=^1.2.3
dependencies.internet-button=*
build.spark-core.environment.CFLAGS=-DSTM32F10X_MD

Overrides

Examples and tests can also contain library.properties files which can override top level one.

Files and directories

Source code structure should be the same as in Arduino libraries. The src dir should be added to search path when compiling allowing users to including libraries with:

#include <InternetButton.h>

Note: removing subdirectory (Library/Library.h) may cause filename clashing (i.e. two libraries have and use utilities.h file). This can be avoided by forcing any private files into subdirectory named after library (i.e. src/Library/utils.h). This should be enforced by command line tools.

Examples should be put in examples dir where each example should be in it's own directory.

Tests should be put in test dir which should contain integration and unit dirs with test cases grouped in their own dirs. Difference here is that integration tests are designed to be run on the device and unit tests should be run on compiling machine.

Example structure

+- doc
+- examples
|  +- example1
|  |  +- example1.ino
|  |  +- library.properties
|  +- example2
|     +- example2.ino
|     +- otherfile.h
+- src
|  +- Library
|  |  +- utils.h
|  +- Library.cpp
|  +- Library.h
+- test
|  +- integration
|  |  +- test1
|  |     +- test1.cpp
|  +- unit
|     +- test2
|        +- test2.cpp
+- library.properties

Similar implementation can be found in uber-library-example.

Dependency resolution

When specifying dependency you target it using npm's semver format.

Whole project can only have one version of a library, so in case of multiple dependencies on library Foo version with highest patch version matching all dependencies will be installed. If such version can't be found installation should fail.

Deep resolution

Possible solution for Scenario B would be to do a deep resolution which would fetch all Bar versions that match and try to select one which depends on Foo in version satisfying rest of dependencies.

Test and example dependencies

Dependencies required just for testing or examples can be specified by test-dependencies and example-dependencies keys in library.properties file.

Architecture targeting

When compiling (which has to happen against specific platform_id) it can be detected if library isn't designed for it (architectures section of library.properties).

#ifdef conditionals are the same as now (and as Arduino's):

Particle firmware defines PLATFORM_ID constant which should be used to define device specific constants (i.e. SPARK_CORE, PARTICLE_PHOTON).

#if defined(ARDUINO_ARCH_AVR)
  // AVR-specific code
#elif defined(ARDUINO_ARCH_SAM)
  // SAM-specific code
#elif defined(PARTICLE)
  // Particle platform specific code
	#if defined(SPARK_CORE)
		// Spark Core specific code
	#elif defined(PARTICLE_PHOTON)
		// Particle Photon specific code
	#endif
#else
  // generic, non-platform specific code
#endif

The same constants can be used when specifying platform overrides in library.properties (i.e. build.$PLATFORM_NAME.*).

Additional constants can be defined using build.$PLATFORM_NAME.define.SYMBOL=VALUE format in metadata.

Enabling debug mode can be done by manipulating build.$PLATFORM_NAME.debug value.

Migrating existing libraries

CLI's library migrate command should help library creators with update. If library is a fork of Arduino one, Particle support should be pushed upstream (original library) if possible.

Library CLI

Build as a separate npm module (particle-library-tools) should expose all methods required for managing libraries. Doesn't expose any binaries/commands by itself. Can be used in Particle Dev to build UI and wrapped around by Particle CLI to provide command line options.

Library Init

Asks user basic info about the library and then scaffolds all necessary files.

Library Install [newlibrary | --save | --deep ]

When nothing was passed installs all libraries from dependencies, example-dependencies and test-dependencies sections of library.properties file. All libraries are installed to cache in ~/.particle/libraries in dirs named :name@:version. Those dirs are then added to search path when compiling project.

When newlibrary was passed just specified library is being installed. --save flag adds installed library to dependencies.

When --deep flag is passed a Deep resolution is performed.

Library Link

Will symlink current library directory ~/.particle/libraries allowing other libraries/apps on current system to use it making library development easier.

Library Search Paths

Echoes paths to libraries from library.properties at specified versions separated by \n. Output should be used to generate LIBRARIES_PATHS env variable (with dirs separated by ;) for gcc.

Library Publish/Unpublish [newversion | major | minor | patch | build]

Publishes/updates library. If passed newversion sets it as current version. If passed major, minor, patch or build bumps specified version component.

Current version is set as git tag and whole local repository is pushed to origin.

Library Doctor

Doctor runs a set of tasks (also available separately):

Lint

Performs:

• static code analysis
• search for non HAL, low level calls
• tries to compile all examples against library
• checks for too broad dependency versions (like *)

Validate

• tests library.properties against schema
• checks if library name is correct and available
• checks if library is in old version and should be migrated

Library Migrate

Automatically converts library from old format to V2.

Library Test

Runs library tests if available. Can be used in CI pipeline.

Build IDE integration

Having almost two hundred libraries seems like a reason to give them more screen real estate than the current drawer. This could share code/UI with Online list of packages.

Additionally UI responsible for adding/removing libraries to an app should modify library.properties file. This file shouldn't be visible to the user (unless it was explicitly requested).

Migration from current

All projects should by parsed by a script which will generate library.properties file from app name and libraries which are used.

Online list of packages

An optional, works more as a showcase of our high compatibility and large database of libraries.

Similar to Atom packages. Consists of:

• libraries index
• library details page
• search functionality

All data would be fetched using API.