touhouwiki-mirror

Small collection of scripts to mirror Doujin CDs raw text and templates from Touhou Wiki for offline querying.

This repository also includes content fetched stored under data/. Files are tracked under Git LFS.

Usage

You will need git-lfs package installed, and run at least one $ git lfs install --skip-repo command before to set up filter commands.

• $ git clone https://git.gammaspectra.live/S.O.N.G/touhouwiki-mirror.git to clone the repository.
• $ ./all.sh to do a full run updating where necessary
• $ ./mirror.sh "<category name>" force if fetching all pages of a category is necessary.
• $ go run . -path data/ for running a server. Opens on port 8777.

Content server features

Small Golang server that loads all albums in an index at startup and is used to serve files/results to queries.

• Access the entire content tree by path
  • Access page by id ex. /pages/54742.wiki
  • Access page by name ex. /pages_by_name/RETRO_FUTURE_GIRLS.wiki
• List files in a directory (ex. /pages/ or /pageindex/Arrangement_CDs/)
• Query database by either any album title or album catalog number
  • Album Title ex. /search?type=album&query=マジコカタストロフィ
  • Album Catalog ex. /search?type=album&query=STAL-1302
  • Values here are normalized so special characters or uppercase are changed. It deals with Unicode too.
  • Additionally, &type=loosealbum exists to match loosely after normalization (will find partial matches).
• Bare CDDB emulator for query and read commands. Used to query by track count + length of album.
  • query cmd ex. /cddb?cmd=cddb+query+730dec08 (no need to provide TOC, but can be provided for more exact match)
  • read cmd ex. /cddb?cmd=cddb+read+Soundtrack54742+730dec08
• Fetch parsed track Lyrics entries
  • ex. /lyrics/Clockup_Flowers
• Fetch album entry by pageid
  • ex. /album/54742

License

Copyright (c) 2022 WeebDataHoarder

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

See the README under data/ for more information about the Touhou Wiki respective content Copyrights and License