Difference between revisions of "Module Development Collaboration"

From CrossWire Bible Society
Jump to: navigation, search
(Version control)
m (Makefile: spaces inserted for alignment)
 
(11 intermediate revisions by 2 users not shown)
Line 3: Line 3:
 
CrossWire will provide to module teams who want to collaborate on a module following tools:
 
CrossWire will provide to module teams who want to collaborate on a module following tools:
  
# A private Git repository - access will be via SSH keys which can act as a scrapbook and collaboration tool
+
# A private Git repository (access will be via SSH keys) which can act as a scrapbook and collaboration tool
 
# Email notification system - all changes to the repository are automatically notified to all collaborators
 
# Email notification system - all changes to the repository are automatically notified to all collaborators
 
# Some templates for the repository and a bunch of scripts to make module making easy.
 
# Some templates for the repository and a bunch of scripts to make module making easy.
  
If you want to get such a repository please email to ''modules'' AT ''crosswire'' DOT ''org''.
+
If you want to get such a repository on the CrossWire server please email to ''modules'' AT ''crosswire'' DOT ''org''.
 +
 
 +
CrossWire itself uses a similar setup for its internal development and for the automatic (re)deployment of modules.
  
 
== Directory structure ==
 
== Directory structure ==
  
ROOT/modtype/language/module/formats/
+
Our [https://github.com/refdoc/Module-tools Github] repository provides makefiles and scripts to setup a module making environment:
 +
 
 +
At the root is a bunch of makefile templates. Choose the appropriate, amend it as necessary and rename it "Makefile" and run
 +
 
 +
  make setup
 +
 
 +
The makefile will create and setup everything to get started. E.g. the makefile for USFM projects will create a structure like this
  
* sword-modules
+
* mods.d   -- SWORD module conf file
** comments
+
* modules  -- compiled module data
** genbook
+
* remotescripts -- scripts pulled in from elsewhere
** lexdict
+
* bin -- locally provided scripts
** texts
+
* osis    -- OSIS docs for osis2mod or sharing with publisher/3rd parties
*** en
+
* usfm    -- USFM for sharing with author or as an intermediate step in conversion
*** fr
 
*** es
 
*** de
 
**** GerLuther
 
**** GerElberfelder
 
***** orig    -- source docs from publisher/author
 
***** fixed   -- source docs after correction of errors, inconsistencies, etc. in text or markup
 
***** docs    -- documentation, correspondence with pubs/authors, permission letters, etc.
 
*****:
 
***** sword  -- compiled Sword module
 
***** gobible -- compiled Go Bible applications (including any smaller collections)
 
***** epub    -- EPUB for publishing to ebooks
 
*****:
 
***** osis    -- OSIS docs for osis2mod or sharing with publisher/3rd parties
 
***** thml    -- ThML for import to Go Bible?
 
***** usfm    -- USFM for sharing with author or as an intermediate step in conversion
 
***** usx    -- USX for sharing with author or as an intermediate step in conversion
 
  
=== Contents of "module" directory ===
+
Depending on the project following directories might be useful.
 +
* gobible -- compiled Go Bible applications (including any smaller collections)
 +
* epub    -- EPUB for publishing to ebooks
 +
* thml    -- ThML for import to Go Bible?
 +
* usx    -- USX for sharing with author or as an intermediate step in conversion
 +
* docs    -- documentation, correspondence with pubs/authors, permission letters, etc.
 +
* orig    -- source docs from publisher/author
 +
* fixed  -- source docs after correction of errors, inconsistencies, etc. in text or markup
  
==== Scripts ====
+
==== bin ====
  
The module directory (e.g. GerElberfelder above) would contain all module-specific scripts necessary for converting source files to modules and other output formats. So general purpose scripts like usfm2osis.pl would not appear here, instead being installed at some system-wide location outside the sword-modules tree.
+
The bin directory (e.g. GerElberfelder above) would contain all module-specific scripts necessary for converting source files to modules and other output formats. So general purpose scripts like usfm2osis.py would not appear here, instead being installed at some system-wide location outside the tree or being pulled into the "remotescripts" directory.
  
 
==== ChangeLog ====
 
==== ChangeLog ====
Line 52: Line 50:
 
==== Makefile ====
 
==== Makefile ====
  
Let's use either GNU make for running scripts or possibly something more platform-independent like a Python or Perl script that calls other scripts. Ideally, we should be able to issue commands like:
+
The makefile templates we offer run usually following
  
  make clean     (delete all output & intermediate format files)
+
  make clean       (delete all intermediate format files)
  make all       (make all supported targets: Sword/OSIS/EPUB/ThML/USFM/...)
+
  make deepclean  (resets everything back to the first step, deleting work directories, intermediate files etc.)
  make sword     (make Sword module plus any necessary intermediate formats, e.g. USFM & OSIS)
+
make setup       (sets up a working environment with relevant directories, pulls in files, scripts etc)
  make epub      (make EPUB module plus any necessary intermediate formats)
+
make all         (create a finalised module ready for publication)
 +
make tests      (run various tests on the files you have produced to find inconsistencies)
 +
  make publish     (push the finalised module out to a staging area and deploy in a repository)
 +
  make localdeploy (copy the finalised module into your own Sword installation to allow easy testing with your own CrossWire biblestudy programmes)
  
 
==== metadata ====
 
==== metadata ====
  
Metadata should be collated into a single file (maybe something XML to facilitate validation for automated generation & parsing) the incorporates all necessary metadata for all of our output formats (OSIS, ThML, TEI, EPUB/MOBI, Sword). Investigate Dublin Core & other metadata standards. Consider developing a set of private-use extensions to an existing standard and consistent methods of encoding all of our relevant .conf values in OSIS/ThML/TEI files.
+
Metadata should be collated into a single file (maybe something XML to facilitate validation for automated generation & parsing) the incorporates all necessary metadata for all of our output formats (OSIS, ThML, TEI, EPUB/MOBI, Sword). Investigate [http://en.wikipedia.org/wiki/Dublin_Core Dublin Core] & other metadata standards. Consider developing a set of private-use extensions to an existing standard and consistent methods of encoding all of our relevant .conf values in OSIS/ThML/TEI files.
  
 
[[Category:Development tools]]
 
[[Category:Development tools]]
 
[[Category:Documentation]]
 
[[Category:Documentation]]
 
[[Category:Modules]]
 
[[Category:Modules]]

Latest revision as of 12:57, 18 December 2015

Version control

. CrossWire will provide to module teams who want to collaborate on a module following tools:

  1. A private Git repository (access will be via SSH keys) which can act as a scrapbook and collaboration tool
  2. Email notification system - all changes to the repository are automatically notified to all collaborators
  3. Some templates for the repository and a bunch of scripts to make module making easy.

If you want to get such a repository on the CrossWire server please email to modules AT crosswire DOT org.

CrossWire itself uses a similar setup for its internal development and for the automatic (re)deployment of modules.

Directory structure

Our Github repository provides makefiles and scripts to setup a module making environment:

At the root is a bunch of makefile templates. Choose the appropriate, amend it as necessary and rename it "Makefile" and run

 make setup

The makefile will create and setup everything to get started. E.g. the makefile for USFM projects will create a structure like this

  • mods.d -- SWORD module conf file
  • modules -- compiled module data
  • remotescripts -- scripts pulled in from elsewhere
  • bin -- locally provided scripts
  • osis -- OSIS docs for osis2mod or sharing with publisher/3rd parties
  • usfm -- USFM for sharing with author or as an intermediate step in conversion

Depending on the project following directories might be useful.

  • gobible -- compiled Go Bible applications (including any smaller collections)
  • epub -- EPUB for publishing to ebooks
  • thml -- ThML for import to Go Bible?
  • usx -- USX for sharing with author or as an intermediate step in conversion
  • docs -- documentation, correspondence with pubs/authors, permission letters, etc.
  • orig -- source docs from publisher/author
  • fixed -- source docs after correction of errors, inconsistencies, etc. in text or markup

bin

The bin directory (e.g. GerElberfelder above) would contain all module-specific scripts necessary for converting source files to modules and other output formats. So general purpose scripts like usfm2osis.py would not appear here, instead being installed at some system-wide location outside the tree or being pulled into the "remotescripts" directory.

ChangeLog

Following the approximate lead of Sword SVN, keep a ChangeLog for each module in the format:

[date in ISO 8601 format: yyyy-mm-dd]\t[contributor name <contributor email>]\n \t[comments]

Makefile

The makefile templates we offer run usually following

make clean       (delete all intermediate format files)
make deepclean   (resets everything back to the first step, deleting work directories, intermediate files etc.)
make setup       (sets up a working environment with relevant directories, pulls in files, scripts etc)
make all         (create a finalised module ready for publication)
make tests       (run various tests on the files you have produced to find inconsistencies)
make publish     (push the finalised module out to a staging area and deploy in a repository)
make localdeploy (copy the finalised module into your own Sword installation to allow easy testing with your own CrossWire biblestudy programmes)

metadata

Metadata should be collated into a single file (maybe something XML to facilitate validation for automated generation & parsing) the incorporates all necessary metadata for all of our output formats (OSIS, ThML, TEI, EPUB/MOBI, Sword). Investigate Dublin Core & other metadata standards. Consider developing a set of private-use extensions to an existing standard and consistent methods of encoding all of our relevant .conf values in OSIS/ThML/TEI files.