Phantom records in the database

[mrysok]

Hi all! Faced with such a problem as phantom entries in the database. To check if I have gone crazy I created a separate page. This post is going to be long, so get ready. What do we have? We have some text (a book, a publication, a contract, etc.) we’ll use the term “article” as a base and store a list of paragraphs.

As you can see from the screenshot, we have 8 paragraphs (let’s remember that). During the editing of this article, the number of paragraphs may change. The functionality of our service implies deleting, editing, and adding new paragraphs with automatic numbering

and paragraph recalculation. During the work it was discovered that some articles did not have the same numbering of paragraphs as their actual number (and sequence). That is, an article of 5 paragraphs could be numbered as follows: 1, 2, 3, 5, 7. Below is a screenshot of a real article with this problem (test page).

Which, of course, is wrong, and I realized that the error is in the work of the process that recalculates these very paragraphs. I created an additional custom process, which is a loop, that recalculated all paragraphs at the press of a button and overwrote the information about the sequence number.

We created a test article with random paragraphs and tested this workflow. The process worked correctly, so I decided to use it to process our problem article. And it didn’t work.

After refreshing the page, everything was back to incorrect. I decided to watch it all through the debug mod and was impressed, upon inspection it turned out that everything worked correctly. But the data in the debugger was different from what was actually happening.

A workflow was created to create a new paragraph. Which simply takes the number of paragraphs from our article and adds + 1 to it.

Such a simple action, but again strange behavior. When you click on the button we add a paragraph with the correct sequence number, but after a moment it spontaneously changes (as if pulling information from a database). I assure you I don’t have any automatic processes in the backend to interact with paragraphs. I want to remind that we have only 8 paragraphs linked to the article in the database (first screenshot).

Then I began to think that perhaps the error in the frontend, and created a loop similar to the first but in the backend.

After checking, nothing changed. I decided to check and change everything manually

The magic of phantom records is back in action. Somehow, miraculously, the server part sees 10 records. I wondered what those entries were, so I set up saving information about them (I also messed up the numbering on purpose for clarity).

I would like to point out that the order of the phantom entries in this article is always identical. That is, the 9th and 5th paragraphs are phantom. If I delete the 2nd one, then of course the offset will be -1.
Then I wanted to try to re-link the entries in the hope that they will be miraculously “lost” in the custom state. But it didn’t happen. When I saved the paragraphs in the custom state, I cleaned up the links and relinked them again, everything stayed the same.
At this point, I don’t know how this bug happens. But I know how to fight it. I have to personally go into the database, delete the link of the first paragraph, save the changes, and then manually link this paragraph to the article again. Then the numbering will normalize.

[tylerboodman]

Do you only have these count issues on Articles where Paragraphs have been deleted previously?

Also you could try taking the list of Paragraphs, sorting by number (descending no) then taking the last item’s number and doing + 1 and seeing it it behaves differently

The “recalculating” of all of them looks like a lot of work, since the numbers don’t necessarily needs to be sequential, just in the order you want the paragraphs.

[keith]

Where do I even begin with this? By saying, “Spoopy!” (But of course spoopy action at a distance isn’t the issue.)

To the OP: When you see things happening “every-other-time-ish” (but fine in step mode) you’re experiencing asynchronous action execution. Which is to say: things you are thinking happen in sequence are not, in fact, happening in sequence.

I won’t help you with this further, but that’s the crux of your issue.

[keith]

OK OK I lied. Here’s a bit more explication on that topic:

FLOPPY: Plugin for localStorage, sessionStorage, IndexedDB storage, List Creation/Manipulation, Iteration, and More! Now with even more video docs!

Oh hai - here’s a new freestyle video where I talk about synchronous vs asynchronous actions (in general) and then demonstrate the new Sync Trigger feature in Floppy. I kind of departed from my intended script here, so I’ll probably do a followup, but this should be helpful to someone… Oh, and here’s the very helpful post I mentioned in the video, wherein Tip: Order of Operation - #2 by aschofer @ashofer from Bubble gave some helpful insights into synchronicity/asynchronicity of Bu…

But I won’t help further. By which I mean I won’t help earlier. By which I mean I won’t help in the correct sequence. Or maybe vice versa?

[mrysok]

1. Yes, this situation occurred only after removal. but not always. This is more likely the first time.
2. I have tried many things)

[mrysok]

Please stop advertising your product under the guise of solving the problem.

[tylerboodman]

One thing I was thinking was just because it’s deleted doesn’t mean it’s removed from that Article’s list of Paragraphs, that is one guess I have is it needs to be removed before deleted.

[mrysok]

You should read this topic more carefully Tip: Order of Operation - #2 by aschofer. Especially the part about the custom workflows I use.

[mrysok]

Of course I know what you mean. But agree, seeing one number of records and actually having another is not what I would like.

[ihsanzainal84]

OP we all understand your current state of frustration in light of your problem but dissing on any form of assistance, regardless if it’s advertising or not, is not gonna get you the help you need.

Both @tylerboodman and @keith have a point. It does look like an async issue that can be resolved with broader understanding of how workflows work.

Those who have been in the forums long enough have read that post you pointed out. I myself have read that and many other posts (official and otherwise) on workflows and async issues in native Bubble. I can tell you from my reading and experience that not everything is covered in documentation or forum posts so don’t be so quick to brush off more resources.

[tylerboodman]

I just mean on creation of a new Paragraph, sort by number and add one to the last one, then if you need to reorder it set it to the average of the two you want. Not like anyone sees this “order” number you can still show their index’s on screen (sequential)

My removal recommendation was regarding that weird thing where Bubble shows (deleted thing) in a field, but if you restored that datatype to an earlier date it magically knows that thing again and it’s spot in the list. I was thinking it’s still in your list as a (deleted thing) and it’s counting that? Just a theory could be completely wrong. So somehow you would synchronously do Make changes to Article → remove Paragraph, then after that do Delete a thing, Paragraph

[mrysok]

I am of the same opinion as you. paragraphs have been deleted and a phantom link to non-existent things is left. But I’ve always understood that it doesn’t make sense to delete paragraphs from an article before deleting them. Deleting an object destroys linking. But most likely the link remains and goes into the count.

[ihsanzainal84]

Looking at your original post again, you are experiencing what i experienced with my app’s form builder and it’s list order. It’s definitely an async issue cause you rely on a lot of additional workflows just to resolve it.

All i did was use List Shifter to update each section’s" list order" field to the current visible list index of each item. 2 workflows on the client-side was all it took.

You can go the non List Shifter route and just match your fields to each paragraph’s cell’s index number in whatever Repeating Group it’s being displayed in.

[mrysok]

I want to make it clear. All of the workflows shown in this post exist on a test page and were created solely to demonstrate the problem. I’m not using a lot of additional workflows just to resolve it.

[mrysok]

I just don’t quite understand how synchronous or asynchronous actions are related to phantom recordings? After all, in fact my post demonstrates their presence as such, right? After all, the action I am displaying is very simple - deleting a record. It’s one step that, for whatever reason, didn’t happen correctly. After which a phantom record was left that messes up another process. And this one is also simple. We create a record with a number (the number of all records in the article +1). So what does this have to do with asynchrony? With the utmost respect for those who do offer advice.

[tylerboodman]

Could try adding a yes/no field called “isDeleted” to your Paragraph (default no), then instead of deleting just set it to “yes”. Then when you go to count your Paragraphs do :filtered with the constraint isDeleted = no so it’s guaranteed excluding “deleted” ones (just as a test)

Again literally just a theory it could be going down the wrong path and it’s just workflows not in sync

[mrysok]

Some smart users think the problem is with the PostgresSQL that Buble uses. Although of course this may not be the case. I saw the problem with another user. He had a problem that all records in PostgresSQL are already indexed and after deleting and rewriting the same rows will have the same id that was before deletion.

[NigelG]

If you are creating things in the backend and then adding them to a list, then they can go missing from that list.

i.e. if you create 100 things by scheduling workflows and as part of that process you add the created thing to a list, you will often end up with 99 things in a list.

If you are deleting something in the back end, and then another process is trying to see if that thing exists, then it can often miscalculate.

i.e. if you delete 100 things from a list of 200, and then do a search expecting 100 … you will often find 101.

No idea why, but Bubble’s backend is essentially inconsistent. It is the way it is.

This will also show up if you are counting things expecting an exact answer. It may miss newly created things, or it may miss newly delete things.

Recursive workflows, and storing counts on control records get you round this.

The fact you see different results when using the debug is a symptom. Slowing things down lets Bubble catch up.

[NigelG]

@mrysok

Some smart users think the problem is with the PostgresSQL that Buble uses. Although of course this may not be the case. I saw the problem with another user. He had a problem that all records in PostgresSQL are already indexed and after deleting and rewriting the same rows will have the same id that was before deletion.

Chances are that Bubble is not using “native” Postgres here. So there is no table on Postgres for your data alone. It is stored as JSON blobs or something.

But yes, it seems like the “index” (be that a Bubble created index or a Postgres one) is sometimes out of step with the actual raw data.

I reported this years ago, and they even had an item on their backlog for database consistency. But that is long gone. The addition of recursive workflows bandaids over that.

[mrysok]

I don’t use the backend to create a paragraph just a simple action (like on deletion).