Difference between revisions of "File Formats"
David Haslam (talk | contribs) (→Miscellaneous: <ref>The VPL file may contain a residue of XML markup</ref>) |
David Haslam (talk | contribs) m (→The SWORD Project Utilities: <BR>) |
||
(223 intermediate revisions by 5 users not shown) | |||
Line 1: | Line 1: | ||
− | + | This page lists some of the more common file formats ''relevant'' to The SWORD Project, associated utilities, and other CrossWire projects. | |
− | + | CrossWire Bible Society respects [[copyright]]. As such, conversion of material that is under copyright without permission from the copyright holders is not supported by The SWORD Project. | |
− | + | == SWORD modules == | |
+ | Other than the source code for the SWORD API, there is no documentation for the file format of a '''SWORD module'''. The intention is that the [[DevTools:SWORD|SWORD API]] (or the [[DevTools:JSword|JSword]] implementation) is used directly or via other language bindings. | ||
− | + | Our module file format is proprietary in the sense that we see no need to document it and certainly no need to stick to it. We change it when we need to. We therefore do not encourage direct interaction with it, but firmly recommend use of the API (either C++ or Java). This is the place where we seek stability and consistency. | |
− | |||
− | + | The SWORD Project supports currently and actively the following markup for module creation: OSIS, [https://tei-c.org/ TEI], ThML and plain text. | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
==The SWORD Project Utilities== | ==The SWORD Project Utilities== | ||
− | Precompiled versions of many of these programs are available in most Linux distributions, using the distribution's package installer. For Windows, they can be found [ | + | Precompiled versions of many of these programs are available in most '''Linux''' distributions, using the distribution's package installer.<BR>For '''Windows''', they can be found [https://github.com/devroles/mingw_sword_package here].<ref>If you have '''Xiphos''' installed in Windows, the Sword utilities are available in the Xiphos\bin folder.</ref><ref>The latest binaries may be found [https://github.com/devroles/mingw_sword_package/releases/tag/1.9.0a here], though currently without cipherraw.exe</ref> |
===Module Creation Tools=== | ===Module Creation Tools=== | ||
− | * imp2gbs | + | It is recommended that Unicode text files used for module creation be [[Encoding|encoded]] as UTF-8.<ref>[http://en.wikipedia.org/wiki/Newline EOLs] should be either Unix style (LF) or Windows style (CRLF). Text files with Mac style EOLs (CR) may give rise to errors or other unexpected behaviour.</ref> |
− | * imp2ld | + | * imp2gbs – imports free-form General books in IMP format to SWORD format [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
− | * imp2vs | + | * imp2ld – imports lexicons, dictionaries, and daily devotionals in IMP format to SWORD format [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
− | * vpl2mod | + | * imp2vs – imports Bibles and commentaries in IMP format to SWORD format [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
− | * [[osis2mod]] | + | * vpl2mod – imports Bibles and commentaries in Verse-Per-Line format to SWORD format [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
− | * xml2gbs | + | * [[osis2mod]] – imports Bibles and commentaries in OSIS format to SWORD format [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
+ | * tei2mod – imports lexicons, dictionaries in TEI format to SWORD format [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] | ||
+ | * xml2gbs – imports free-form General books in OSIS or ThML format to SWORD format [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] | ||
===Diagnostic Tools=== | ===Diagnostic Tools=== | ||
− | * mod2imp | + | * mod2imp – creates an IMP file<ref>The IMP file may contain a residue of XML markup</ref> from an installed module [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
− | * | + | * emptyvss – exports a list of verses missing from the module (useful for testing modules during development) [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
− | |||
− | === | + | ===Legacy format conversion Tools=== |
− | * gbf2osis.pl | + | * gbf2osis.pl – a PERL utility for converting GBF to OSIS [http://crosswire.org/ftpmirror/pub/sword/utils/perl/gbf2osis.pl †] |
− | * step2vpl | + | * step2vpl – export a STEP book in Verse-Per-Line (VPL) format [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
* [[DevTools:Misc#thml2osis|thml2osis]] - converts ThML to OSIS format. | * [[DevTools:Misc#thml2osis|thml2osis]] - converts ThML to OSIS format. | ||
− | |||
===OSIS Utilities=== | ===OSIS Utilities=== | ||
− | * | + | * vs2osisref – returns the osisRef of a given (text form) verse reference [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
− | + | * xml2gbs – imports free-form General books in OSIS or ThML format to SWORD format [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] | |
− | * xml2gbs | ||
===Miscellaneous=== | ===Miscellaneous=== | ||
− | * cipherraw | + | * cipherraw – used to encipher SWORD modules [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
− | * diatheke | + | * [[Frontends:Diatheke|diatheke]] – a basic CLI SWORD front-end [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
− | * | + | * [[mkfastmod]] – creates a search index for a module<ref>Aside: To create a list of installed modules with descriptions, enter the following command, optionally redirecting stderr to a log file.<pre>mkfastmod /? 2>mkfastmod.log</pre></ref> [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
− | * mod2zmod | + | * [[mod2zmod]] – creates a compressed module from an installed module [http://crosswire.org/ftpmirror/pub/sword/utils/win32 †] |
− | + | ==== Notes on SWORD Tools ==== | |
− | |||
− | === Notes === | ||
<references /> | <references /> | ||
===Recommended Non-SWORD Utilities=== | ===Recommended Non-SWORD Utilities=== | ||
− | * uconv | + | * uconv – a utility from [http://icu-project.org/ ICU] for converting between various character encodings, perform normalization, transliterate texts, etc. (It's similar to iconv, but much, much more powerful.) uconv.exe is part of the [http://crosswire.org/ftpmirror/pub/sword/utils/win32 sword utilities] |
− | * xmllint | + | * xmllint – a utility (part of the [http://xmlsoft.org/ libxml2] distribution) for validating XML documents [http://crosswire.org/ftpmirror/pub/sword/utils/win32 *] |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | == | + | ==Formats for which CrossWire maintains converters== |
− | + | The SWORD Project uses primary source e-texts. These texts come in numerous formats. CrossWire maintains converters for a number of formats, described below. The converters may target other markup formats, e.g. TEI or OSIS, or may simply export binary data to text, as is the case with our STEP exporter. Specific discussion of each of the available converters is found elsewhere on this page. | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
===USFM=== | ===USFM=== | ||
− | [http:// | + | [http://paratext.org/usfm ''Unified Standard Format Markers''] |
− | This plain-text format is a common internal-use format within Bible translation agencies and Bible societies. It is the native format of [http://paratext | + | This plain-text format is a common internal-use format within Bible translation agencies and Bible societies. It is the native format of [http://paratext.org/ ParaTExt]. Paratext is used by more than 60% of all Bible translators world-wide. The current release is [https://pt8.paratext.org/ ParaTExt 8.0]. |
− | + | Though '''USFM 2.4''' suffices for most Bibles, [https://ubsicap.github.io/usfm/ USFM 3.0] is now available and has several new features. The standard is open source and is maintained at [https://github.com/ubsicap/usfm ubsicap/usfm]. | |
− | + | CrossWire now has a Python script called usfm2osis.py<ref>This replaces our earlier Perl script [http://crosswire.org/ftpmirror/pub/sword/utils/perl/usfm2osis.pl usfm2osis.pl].</ref> which converts USFM to OSIS for subsequent import to SWORD's native format. See [[Converting SFM Bibles to OSIS]]. | |
− | + | USFM uses a separate file for each Bible book. USFM is also supported by the open-source program called [http://bibledit.org/ Bibledit]. There are examples of Bibles in USFM format available for download at [http://ebible.org/]. These include the [http://ebible.org/bible/kjv/kjvsf.zip KJV], [http://ebible.org/bible/asv/asvsf.zip ASV], and [http://ebible.org/bible/web/websf.zip WEB] Bibles. | |
− | |||
− | + | USFM is one of the formats that can be used by [[Projects:Go Bible/Go Bible Creator|Go Bible Creator]]. | |
− | + | '''Note:''' | |
− | '' | + | <references /> |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | ' | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
==Other Utilities== | ==Other Utilities== | ||
Line 198: | Line 72: | ||
===Go Bible utilities=== | ===Go Bible utilities=== | ||
− | * Go Bible Creator - a Java SE program for converting either ThML or OSIS or USFM to Go Bible | + | * [[Projects:Go Bible/Go Bible Creator|Go Bible Creator]] - a Java SE program for converting either ThML or OSIS or USFM to [[Projects:Go Bible|Go Bible]]. It is being enhanced by SIL to be capable of converting source text in [[File Formats#XHTML|XHTML-TE]] format. |
* [http://gbcpreprocessor.codeplex.com/ Go Bible Creator USFM Preprocessor] – This is a tool to parse through and identify, correct and publish USFM file formats into a file format that can easily be put into the Go Bible mobile phone program. | * [http://gbcpreprocessor.codeplex.com/ Go Bible Creator USFM Preprocessor] – This is a tool to parse through and identify, correct and publish USFM file formats into a file format that can easily be put into the Go Bible mobile phone program. | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
===ThML Utilities=== | ===ThML Utilities=== | ||
* CCEL Desktop - a program for viewing and developing CCEL books [http://ccel-desktop.sourceforge.net/] | * CCEL Desktop - a program for viewing and developing CCEL books [http://ccel-desktop.sourceforge.net/] | ||
− | == | + | == See also == |
− | * | + | * [[DevTools:IMP Format|IMP Format]] – general import format used for various module types |
− | + | * [[DevTools:GBF|General Bible Format (GBF)]] – legacy format now deprecated | |
− | + | * [[DevTools:ThML|Theological Markup Language (ThML)]] – legacy format now deprecated | |
− | [[ | + | * [[Frontends:Bookmarks Standard]] |
− | + | * [[File Formats Cruft]] | |
− | |||
− | |||
− | * [ | ||
− | |||
+ | [[Category:Development tools]] | ||
+ | [[Category:File formats]] | ||
[[Category:OSIS]] | [[Category:OSIS]] | ||
[[Category:ThML]] | [[Category:ThML]] | ||
[[Category:Utilities]] | [[Category:Utilities]] | ||
+ | [[Category:USFM]] | ||
+ | [[Category:Unicode]] | ||
+ | [[Category:Bibledit]] | ||
+ | [[Category:Paratext]] |
Latest revision as of 12:11, 20 February 2021
This page lists some of the more common file formats relevant to The SWORD Project, associated utilities, and other CrossWire projects.
CrossWire Bible Society respects copyright. As such, conversion of material that is under copyright without permission from the copyright holders is not supported by The SWORD Project.
Contents
SWORD modules
Other than the source code for the SWORD API, there is no documentation for the file format of a SWORD module. The intention is that the SWORD API (or the JSword implementation) is used directly or via other language bindings.
Our module file format is proprietary in the sense that we see no need to document it and certainly no need to stick to it. We change it when we need to. We therefore do not encourage direct interaction with it, but firmly recommend use of the API (either C++ or Java). This is the place where we seek stability and consistency.
The SWORD Project supports currently and actively the following markup for module creation: OSIS, TEI, ThML and plain text.
The SWORD Project Utilities
Precompiled versions of many of these programs are available in most Linux distributions, using the distribution's package installer.
For Windows, they can be found here.[1][2]
Module Creation Tools
It is recommended that Unicode text files used for module creation be encoded as UTF-8.[3]
- imp2gbs – imports free-form General books in IMP format to SWORD format †
- imp2ld – imports lexicons, dictionaries, and daily devotionals in IMP format to SWORD format †
- imp2vs – imports Bibles and commentaries in IMP format to SWORD format †
- vpl2mod – imports Bibles and commentaries in Verse-Per-Line format to SWORD format †
- osis2mod – imports Bibles and commentaries in OSIS format to SWORD format †
- tei2mod – imports lexicons, dictionaries in TEI format to SWORD format †
- xml2gbs – imports free-form General books in OSIS or ThML format to SWORD format †
Diagnostic Tools
- mod2imp – creates an IMP file[4] from an installed module †
- emptyvss – exports a list of verses missing from the module (useful for testing modules during development) †
Legacy format conversion Tools
- gbf2osis.pl – a PERL utility for converting GBF to OSIS †
- step2vpl – export a STEP book in Verse-Per-Line (VPL) format †
- thml2osis - converts ThML to OSIS format.
OSIS Utilities
- vs2osisref – returns the osisRef of a given (text form) verse reference †
- xml2gbs – imports free-form General books in OSIS or ThML format to SWORD format †
Miscellaneous
- cipherraw – used to encipher SWORD modules †
- diatheke – a basic CLI SWORD front-end †
- mkfastmod – creates a search index for a module[5] †
- mod2zmod – creates a compressed module from an installed module †
Notes on SWORD Tools
- ↑ If you have Xiphos installed in Windows, the Sword utilities are available in the Xiphos\bin folder.
- ↑ The latest binaries may be found here, though currently without cipherraw.exe
- ↑ EOLs should be either Unix style (LF) or Windows style (CRLF). Text files with Mac style EOLs (CR) may give rise to errors or other unexpected behaviour.
- ↑ The IMP file may contain a residue of XML markup
- ↑ Aside: To create a list of installed modules with descriptions, enter the following command, optionally redirecting stderr to a log file.
mkfastmod /? 2>mkfastmod.log
Recommended Non-SWORD Utilities
- uconv – a utility from ICU for converting between various character encodings, perform normalization, transliterate texts, etc. (It's similar to iconv, but much, much more powerful.) uconv.exe is part of the sword utilities
- xmllint – a utility (part of the libxml2 distribution) for validating XML documents *
Formats for which CrossWire maintains converters
The SWORD Project uses primary source e-texts. These texts come in numerous formats. CrossWire maintains converters for a number of formats, described below. The converters may target other markup formats, e.g. TEI or OSIS, or may simply export binary data to text, as is the case with our STEP exporter. Specific discussion of each of the available converters is found elsewhere on this page.
USFM
Unified Standard Format Markers
This plain-text format is a common internal-use format within Bible translation agencies and Bible societies. It is the native format of ParaTExt. Paratext is used by more than 60% of all Bible translators world-wide. The current release is ParaTExt 8.0.
Though USFM 2.4 suffices for most Bibles, USFM 3.0 is now available and has several new features. The standard is open source and is maintained at ubsicap/usfm.
CrossWire now has a Python script called usfm2osis.py[1] which converts USFM to OSIS for subsequent import to SWORD's native format. See Converting SFM Bibles to OSIS.
USFM uses a separate file for each Bible book. USFM is also supported by the open-source program called Bibledit. There are examples of Bibles in USFM format available for download at [1]. These include the KJV, ASV, and WEB Bibles.
USFM is one of the formats that can be used by Go Bible Creator.
Note:
- ↑ This replaces our earlier Perl script usfm2osis.pl.
Other Utilities
These are not part of The SWORD Project, but may be useful. A link is given for each.
Go Bible utilities
- Go Bible Creator - a Java SE program for converting either ThML or OSIS or USFM to Go Bible. It is being enhanced by SIL to be capable of converting source text in XHTML-TE format.
- Go Bible Creator USFM Preprocessor – This is a tool to parse through and identify, correct and publish USFM file formats into a file format that can easily be put into the Go Bible mobile phone program.
ThML Utilities
- CCEL Desktop - a program for viewing and developing CCEL books [2]
See also
- IMP Format – general import format used for various module types
- General Bible Format (GBF) – legacy format now deprecated
- Theological Markup Language (ThML) – legacy format now deprecated
- Frontends:Bookmarks Standard
- File Formats Cruft