Using BERT to mine large datasets for purchasing sentiment
This is a quick real-world experiment with computer-based Natural Language Processing, demonstrating how it can be of use outside the very large corporations which have been driving its rapid development in the last few years.
Aim
The aim is to analyse the Bicycle Stack Exchange site to extract information of what the posters (in aggregate, no identity at all is used) may be interested in buying. In the dataset there are a total of 58,000 posts which could be manageable for a human to get a feel for in a day or two, but the aim is to develop an automatic way of summarising the sentiment which could be applied to a much larger dataset. A significant complicating factor of this dataset is that the majority of posts are not about buying at all hence understanding significance of processed output is important.
An analysis such as this could, for example, be used by a new manufacturer to identify potential products; or it could be used by physical shops to identify which new types of products could be of interest to cyclists.
Method
I use the Bidirectional Encoder Representations from Transformers (BERT) large pre-trained model as available in the Python transformers package.
Each post on the Bicycle stack exchange is analysed in turn with the
model transformers.BertForQuestionAnswering : the post itself is
the input paragraph and the question in this case was simply: What
should I buy?1 . To each answer a significance from the model is
attached. The Bicycle stack exchange posts data can be download from
e.g. archive.org
.
Dataset analysis is done using a Tesla K80 GPU and Google Colab – processing on even a high-end high core count CPU is substantially slower.
Results
The top 200 results (ordered by significance) are shown below. They illustrate nicely both the limits and successes of computer-based Natural Language Processing:
-
A significant fraction of the answers are not really answers to the question “What should I buy?” and not relevant to an analysis of purchasing sentiment – so the False Positive rate in this particular instance is significant
-
There is however a great deal of useful information, in a very compact form – this is a summary of almost 8 million words of input data! Yet it is possible to scan the summary in less than five minutes and get a very good idea of what the posters are thinking of purchasing.
-
In fact, even as a casual cyclists unconnected to the industry this NLP summary has given me some good ideas of what to buy (in case you are interested: “squirt dry lube”, “bungee cords”, “pedal extenders” – I had no idea this existed!). I’ll be giving the fixed gear a miss though.
All the results:
| Answer | Score |
|---|---|
| without replacing the cranks | 3.984334 |
| brake lever | 3.987443 |
| a multispeed wheel | 3.992327 |
| single - speed bike | 3.998377 |
| kickstand has been designed for those heavy loads | 4.000814 |
| 6 - bolt or centre - lock rotor interface | 4.002281 |
| aluminum | 4.007900 |
| a single wheel bike trailer | 4.007934 |
| chris king or cane creek | 4.013450 |
| my bike is a racing bike - or - my bike has a … | 4.014205 |
| a 2 - wheeled cargo trailer | 4.014745 |
| cavitation | 4.016710 |
| 1997 | 4.024745 |
| 25mm | 4.036997 |
| your bike does not fit properly | 4.037504 |
| shoes legwear ( rain ) coats face / helmet | 4.044062 |
| new asphalt or roadworks | 4.054311 |
| to get a really thick pair of gloves | 4.055343 |
| solvent | 4.067034 |
| kryptonite or abus thx | 4.067524 |
| 30 seconds | 4.067729 |
| 700x20 - 25c or 700x25 - 32c | 4.069897 |
| it is advantageous | 4.071337 |
| it can cause the chain to hit the chainstay an… | 4.072789 |
| bags for cell phones or very slim compact cameras | 4.084034 |
| sti levers | 4.087434 |
| a cheap pair of platform pedals | 4.087685 |
| low likelihood of forgetting you pack on your … | 4.087863 |
| a trailer or rack | 4.087946 |
| kickstarter kit by barak electric | 4.089828 |
| stabilising the fork so it does not wobble to … | 4.092712 |
| bicycle tow bar | 4.093565 |
| a little cylinder that runs on the front brake… | 4.094686 |
| to repaint | 4.097400 |
| 9sp chainrings | 4.112884 |
| there is some personal preference from mechani… | 4.121975 |
| what lock | 4.133030 |
| use your front brake and push your rear end | 4.141050 |
| a new bike | 4.142982 |
| cateye micro wireless mc100w or cateye velo 7 | 4.148888 |
| ~ $ 480 | 4.149406 |
| magura | 4.158324 |
| etrto code | 4.165245 |
| 319 pounds | 4.168193 |
| tacx blue or satori | 4.171396 |
| 24 speed carrera vengeance | 4.176042 |
| where would be the best place to look for this… | 4.182470 |
| a trailer bike | 4.182586 |
| google maps | 4.188448 |
| mountain bike | 4.190742 |
| backwards | 4.196258 |
| blackberry | 4.198652 |
| i have no idea where it’s from | 4.202947 |
| most stores only offer trailers sized for chil… | 4.208519 |
| training | 4.211099 |
| 700x25c | 4.211437 |
| 5, 000 miles | 4.214664 |
| strength or cardiovascular goals | 4.219513 |
| faster than their eventual main pace | 4.228981 |
| token tf24 or the token ninja tf37 | 4.240726 |
| in a bladder | 4.254608 |
| a wheel | 4.254874 |
| mechanical | 4.256982 |
| a hex bolt | 4.265576 |
| beater steel | 4.279584 |
| in the spokes. if you don’t have a pannier or … | 4.280002 |
| mountain bike | 4.290862 |
| quite a bit | 4.295190 |
| rims | 4.297310 |
| pulling up or pushing down | 4.298195 |
| a helicoil kit | 4.301330 |
| if parking your own bike in that space makes i… | 4.305521 |
| rims that will fit cyclocross or road bike tires | 4.311843 |
| disk brakes | 4.312477 |
| screw clamps | 4.333948 |
| helmet mounted lights | 4.340793 |
| 14 / 2mm | 4.341318 |
| new / better brakes | 4.364946 |
| back brake | 4.375150 |
| corrosion proof coating | 4.381238 |
| price, capacity and design | 4.382625 |
| mt - 40 | 4.383696 |
| NaN | 4.384270 |
| cn - m981 cn - m980 cn - hg95 cn - hg94 cn - h… | 4.391233 |
| rest | 4.400147 |
| ergonomic “ locking grips | 4.401465 |
| replacing your old tire with another tire you … | 4.403919 |
| the external width just keeps shrinking | 4.409768 |
| torque wrench | 4.424376 |
| increases bb stiffness and durability, while r… | 4.433124 |
| too narrow for optimal grip for most people | 4.436118 |
| non - standard | 4.455884 |
| rotary plumbing pipe cutter | 4.458583 |
| a single speed bike | 4.477700 |
| a standard pull brake lever | 4.480142 |
| replacement shifters | 4.493139 |
| how would i know | 4.506325 |
| NaN | 4.506698 |
| crank arm | 4.512474 |
| extremely functional | 4.516920 |
| superglue or duct tape | 4.522046 |
| chain tensioners | 4.524573 |
| a really long seat post installed way too far up | 4.536146 |
| single spokes | 4.539762 |
| rear metal part of the saddle | 4.546989 |
| a crank tool | 4.600437 |
| flinging my leg over the saddle while the bike… | 4.603375 |
| a “ boot “ inside the tire | 4.617736 |
| a upright conventional bicycle | 4.639482 |
| advertising | 4.659174 |
| wired vs wireless | 4.659637 |
| woods ( dunlop ) valve | 4.664250 |
| high - end | 4.664501 |
| designating some car lanes as bike - only lanes | 4.665493 |
| funds and storage | 4.668933 |
| any tire of similar quality would be no more o… | 4.677284 |
| the distance between the tire and the fender | 4.683753 |
| vibrations that hurt my fingers | 4.686222 |
| mechanical or hydraulic | 4.686745 |
| a helmet with a face shield | 4.703397 |
| nowadays we don’t care about scratches “. but … | 4.707049 |
| hed tomcat disc | 4.710005 |
| a 54cm bike | 4.715860 |
| every 100km to 200km | 4.716311 |
| presta | 4.725976 |
| a bicycle | 4.728568 |
| kool stop thinlines | 4.735795 |
| they are lighter and smaller than bikes easier… | 4.736367 |
| double sided tape and some sort of neoprene or… | 4.748918 |
| sram / avid | 4.751222 |
| ride up grades | 4.760292 |
| chain stay length | 4.768253 |
| wider | 4.779448 |
| improving your tt or climbing speeds | 4.779812 |
| medium sandpaper | 4.779857 |
| 700c | 4.802800 |
| don’t | 4.817946 |
| in a hydration pack on my back | 4.842426 |
| cartridge bottom brackets | 4.849416 |
| ball and rod | 4.866506 |
| day / night, foggy vs. direct sun, perhaps the… | 4.872999 |
| many other companies manufacture freehubs with… | 4.891198 |
| custom pedal extenders | 4.920067 |
| it could very easily sap the energy reserves f… | 4.922484 |
| your preferences and what kind of riding you w… | 4.924358 |
| maximum braking efficiency and better control … | 4.931591 |
| tubeless tyres | 4.950227 |
| disc brakes | 4.963746 |
| i can’t find info | 4.970236 |
| my bike is very sturdy | 4.973813 |
| ” brake hard, then release completely & amp ; … | 4.974698 |
| degreaser | 4.983543 |
| ebay | 4.986038 |
| it depends | 5.017685 |
| 60 mm between the jockey wheels | 5.028659 |
| as far forward | 5.051633 |
| sunblock | 5.114271 |
| $ 1500 | 5.120024 |
| 700c | 5.128308 |
| rolling the hips forward slightly | 5.159712 |
| 26. 8 mm | 5.160173 |
| hot water | 5.165559 |
| a simple conversion | 5.168078 |
| castrol ep 90 | 5.184137 |
| bungee cords | 5.201746 |
| the spinning mass of the wheel | 5.233724 |
| hardtail 29er | 5.325470 |
| paint job | 5.327999 |
| 5 % less | 5.331928 |
| schraeder | 5.334766 |
| nut driver | 5.349661 |
| 20 + hours a week | 5.357866 |
| local bike shops | 5.364977 |
| hybrid | 5.377503 |
| grit | 5.387483 |
| toasty | 5.391614 |
| puncture repair glue | 5.401884 |
| bike trousers | 5.405810 |
| non - park tool tool | 5.408729 |
| comfort, grip and possibly rolling resistance | 5.426383 |
| fixed gear | 5.435682 |
| internal - cam | 5.438887 |
| pinkbike | 5.464226 |
| watts | 5.466736 |
| a tarp | 5.500462 |
| merinos | 5.509621 |
| squirt dry lube | 5.529767 |
| within a couple feet | 5.539470 |
| does not drop as far into potholes, and genera… | 5.545146 |
| 700x28c | 5.551797 |
| machines generally do a more precise job than … | 5.557894 |
| 35 mph | 5.581415 |
| one speed chain | 5.587879 |
| a riding position that’s optimized for hard pe… | 5.757022 |
| much more rapid | 5.757543 |
| signage on the bike path saying bicycles are a… | 5.778970 |
| between dropout and skewer end | 5.820508 |
| new parts or adjustments | 5.843324 |
| autumn | 5.848872 |
| metal rod brakes | 5.877050 |
| a pressure gauge | 5.978394 |
| 2000 miles | 6.056773 |
| 700x35c | 6.087281 |
| 56mm | 6.097930 |
| off - road or on dirty / wet roads | 6.140635 |
| change clothes at work | 6.186455 |
| because of the difference in distance travelled | 6.192146 |
| longest battery life | 6.230284 |
| shifting gears | 6.242918 |
| by using it | 6.256811 |
| hub, hub axis, bearings, spokdes | 6.356838 |
| 100 % daylight ; no mechanical failures ; and,… | 6.378467 |
| safety reasons | 6.620848 |
| overactive | 6.975286 |
| melted down and the metal used for something new | 7.1861 |
Updates
- Update 1 Aug 2021 with link to data
-
Experimenting with different questions is informative but outside the scope of this initial post! ↩