Reverse engineering the agency / RFP match system

[georgecollier]

I’ve done some digging/reverse engineering, and am happy to be able to share how agency matches are calculated in the RFP portal.

There are 6 measurements that are used to calculate the score. The agencies with the top score are shown.

The maximum possible score is 6.

When I made an RFP request, my request was scored for a match against 109 agencies. There are 267 agencies registered on the agencies area of Bubble’s site, and there are 1,070 in Bubble’s database. How is it decided which agencies are scored for matching? Who knows. I’d guess not all of the 267 have enabled RFPs, but 109 seems a bit low. Perhaps any agencies that don’t want my budget because it’s too low are filtered out, but that makes the budget score we’ll explore irrelevant!

My RFP

I set my RFP budget to $10k-$20k, language to English, preference to Agency, and chose a handful of services the agency must require.

Budget

If your RFP’s budget is greater than the agency’s minimum budget, this value is a 1. Else, it scores 0.

Developer preference

If this agency’s type (freelance / agency) is the same as your request, it scores 1. Else, it scores 0.

Language

If this agency has the language you specify in your RFP, it scores 1. Else, it scores 0.

Project size

If this agency accepts projects of your size. Now, I was unable to work out how this is calculated. My RFP was only ever classified as ‘small’, no matter what timeline or budget I set. It seems that this value is redundant and all RFPs on portal are classified as small.

There are some agencies that have selected to not take small projects. If my hypothesis above is true, then they’re not receiving any RFPs.

Project type

Either new build or ongoing app development. If the agency has opted to receive the type you’re looking for, this is scores as 1. Else it is scored as zero.

Services offered

You can select multiple services you want your agency to offer. This score can range from 0 to 1. It is calculated as:

number of services offered by agency that you request / number of services you requested.

In other words, what percentage of your requested services (e.g product design, mobile app development) can be fulfilled by the agency.

34 agencies got the top score

31% of the agencies that were scored were tied for 1st place. That means that the agencies shown must’ve been randomly selected from these. So, it is a bit of a luck game, but there’s probably not a fairer way to do it.

How it works

Behind the scenes, Bubble creates an ‘RFP Suggestion’ Thing for every agency it tests against. This calculates the necessary score parameters. When the user sees the matches loading, Bubble is creating 100+ Things in the front-end to find out which score the highest. You can of course see this information in the developer console, which is where my data comes from.

Hope it helps! I’ll be publishing some analysis on the Bubble agency landscape shortly too.

[chris.williamson1996]

Ooooo you went and played and found the ridiculous amount of data they leak on agencies

Maybe someone should tell bubble that the full list of all an agencies apps shouldn’t be leaked publicly🤷‍♂️ @fede.bubble

This is a beautiful right up👏

[lindsay_knowcode]

Wow that is strong analysis. Kudos.

[fede.bubble]

Hey @georgecollier

Great share. I wanted to touch on a couple of things:

• Matching system: yeah, you nailed it. The team shared info with agencies on how this is calculated in the past, but it wasn’t widely shared so I understand if this is the first time most users hear about it. The team told me the way these RFPs get processed might change in the future: it’s all about making sure we do a good job matching users with agencies.
• Available information: The data you found in the dev console is publicly accessible (It wasn’t by accident). I don’t want users reading this and to think Bubble had some major privacy mishap. For example, you could go to /agencies and aggregate info like min budget, hourly rates, countries, etc.

So yeah! Great work putting this together. I hope nobody freaks out thinking Bubble is super unsafe or something

PS: I would also hope that if anyone came across some major vulnerability that they would reach out to the team first, rather than post publicly, and follow responsible disclosure practices. At the end of the day, we all want to keep Bubble safe and running

[georgecollier]

@fede.bubble

it’s all about making sure we do a good job matching users with agencies.

Yes, I think it’s fair and can’t think of anything better. I also can’t think of a way to exploit it, which is a shame for me, but good for the system

[fede.bubble]

@georgecollier

I also can’t think of a way to exploit it,

pls don’t exploit anything, I just started here

[boston85719]

@fede.bubble

it’s all about making sure we do a good job matching users with agencies.

I hope agencies is being used as a catch all for agencies and freelancers

[georgecollier]

Agency = anyone with agency plan that’s approved by Bubble

[dev113]

Nice analysis!
Do you have an idea what does the number 1,070 actually represent? is it a total number of agencies ever existed compared to 267 currently active?

[bubble62]

Damn, this is so impressive! Kinda sad that it would somewhat come down to your luck no matter how good you play your cards. Appreciate the info tho, great help.

[boston85719]

@georgecollier

I also can’t think of a way to exploit it, which is a shame for me, but good for the system

@georgecollier

If your RFP’s budget is greater than the agency’s minimum budget, this value is a 1. Else, it scores 0.

Set a low budget, get more matches, but downside is get matched to rfp requests that might not be as valuable as actually looking for

@georgecollier

Developer preference

If this agency’s type (freelance / agency) is the same as your request, it scores 1. Else, it scores 0.

Two Agency accounts, one for the Agency and the second for the Freelancer, this way the agency will show up for either, and since freelancer is only a single person, it is just another $80/month (or whatever the agency account is these days).

@georgecollier

Language

If this agency has the language you specify in your RFP, it scores 1. Else, it scores 0.

Select every language…use any AI to translate communications

@georgecollier

Project size

If this agency accepts projects of your size. Now, I was unable to work out how this is calculated. My RFP was only ever classified as ‘small’, no matter what timeline or budget I set. It seems that this value is redundant and all RFPs on portal are classified as small.

There are some agencies that have selected to not take small projects. If my hypothesis above is true, then they’re not receiving any RFPs.

Accept all project sizes, but downside is you get RFP that are not really of the size the agency might actually be looking for.

@georgecollier

Project type

Either new build or ongoing app development. If the agency has opted to receive the type you’re looking for, this is scores as 1. Else it is scored as zero.

Select both, most agency account holders likely already are

@georgecollier

Services offered

You can select multiple services you want your agency to offer. This score can range from 0 to 1. It is calculated as:

number of services offered by agency that you request / number of services you requested.

In other words, what percentage of your requested services (e.g product design, mobile app development) can be fulfilled by the agency.

Offer them all, as most agency account holders likely are already, and when you get matched and receive an RFP for some service the agency is not capable of offering, either do not reply, or make the sale and sell the project to a freelancer. I believe a lot of agencies out there utilize freelancer very often, and get an opportunity to sound large and as an international agency by describing the different countries they work in. To me, that makes for some not so good apps being put out by agencies using freelancers. I can think of only a few that do it right and only use in-house developers.

Also, gaming the system for becoming a Gold Tier agency (ie: larger agency) is advantageous, as Bubble mentioned at least one Gold Tier Agencies is almost always shown.

@grace.hong

We always try to include one Gold agency partner in the top three matches (assuming they meet the users’ criteria). This is why you might see a Gold agency in your match results even if you selected freelancer for your developer preference.

I think to do that, one approach is to just pay for more developer accounts and take the certification exam for each you add. At $80/month for each, it might be an economically advantageous thing to do if it means getting more RFPs that could be for some larger budget app builds.

@bubble62

Kinda sad that it would somewhat come down to your luck no matter how good you play your cards.

Success is just being prepared, having worked hard (ie: play your cards well) to strike when you have your lucky moment. I used to play Texas Hold’em quasi professionally (it is how I paid for university tuition and living expenses), and one of the most important lessons I can take from playing and winning poker tournaments, is patience is key, being better than others is necessary, but luck is essential and the only thing that will make or break your success.