Parsing and storing <generator> elements

I’m interested in some creating some functionality based around the tag that some feeds have. RSS 2.0 and Atom both describe this optional element. For instance Wordpress feeds will typically have some information in here denoting that they are coming from Wordpress, and usually a version too.

One assumption I am making is that this data is not useful in TTRSS itself and that a patch to add parsing and storing it would be skeptically received. A quick investigation leads me to think that such a patch would require a schema addition of a new column in ttrss_feeds, and an addition to SimplePie to include generator element parsing (in library/SimplePie/Source.php). Do let me know if this would be a welcome patch though.

So if I need to add this as a plugin, I think I could use HOOK_FEED_PARSED if I got generator added to SimplePie. Or HOOK_FEED_FETCHED if I chose to just parse the field out myself.

To store the data, I think I would need to use ttrss_plugin_storage. Name would be some string unique to my plugin (generated_store or something), and content would probably contain a key and a value maybe in the format : so a real example would be 13: Worpressv4

This is not ideal but I think queries that look for a row where the content column starts with a string representation of an integer will be mostly OK to work with.

Any thoughts overall?

there’s no simplepie in tt-rss. not for many years now.

you can write a plugin which would add whatever columns you need and work on them accordingly via hooks.

you can either extend one of the tables with additional columns (probably not the best idea because i won’t be aware of it and some further update might break things for you) or make a separate table to store your data - i would recommend this.

plugin storage is for simple key/value storage for stuff like plugin settings, it won’t be performant enough to store any amounts of feed-related data and it is de/serialized into RAM on startup so you’re going to run into memory limits.

you can provide a separate sql file to create your tables bundled with the plugin or autocreate necessary tables in plugin init if necessary.

https://git.tt-rss.org/fox/ttrss-perceptual-image-hash/src/master/sql/phash_schema_pgsql.sql

i suggest naming your tables similarly to this so it would be easier to figure out which table relates to which plugin

if you make your own table relying on ttrss_entries etc don’t forget to add on delete cascade foreign keys so that there’s no leftover data for purged articles.

OK that is very helpful, thanks. I will create my own table for the plugin, named as you suggest. Perhaps I will store a DB schema version in the plugin storage table if I get really crazy.

So I have started to create my plugin and I to the part of creating the database table. Looking at your example plugin fox/ttrss-perceptual-image-hash, I don’t see any code that runs the SQL code in the sql/ folder. And when I install and enable that plugin I don’t see that table getting created in my database. And finally, the documentation for how to install the plugin does not describe that someone needs to run the SQL code manually. So, what is the best way for a plugin to create a table?

yeah, schema has to be installed manually for that plugin. i seem to vaguely recall a different (?) plugin which did something with tables on init but i couldn’t find it when i was writing my above post.

now that i think about it checking for tables all the time (init would be called on every http request) is not the best idea even though it probably won’t have a tangible effect on performance. so maybe simply having a separate user-installed SQL file is better.

also, plugin README should definitely mention schema installation, it’s an oversight on my part. i’ll make a note to update the README. i guess i thought it would be obvious. :man_shrugging:

OK I have created my plugin with some basic functionality.

I did figure out what I think is a reasonable way to automatically set up the database. When the preferences screen is loaded it does the database checking/creation/updating. It shows some info about that on it’s own accordion panel. Schema version is stored in the ttrss_plugin_storage table…I found the very nice get() and set() functions that for that, thank you!

it’s probably not the best idea, depending on how you actually implemented it. this table is for per-user plugin settings which may be cleared by user at any time. also, plugins enabled for multiple users may have different data in there.

it would be better to keep it in the database properly, maybe in a separate table.