The National Police are looking for a man who took packets of hashish from a bundle that washed up on the beach in Arenal. Around 1pm, beachgoers watched on in astonishment as he cut the bundle open, removed some of the contents and then ran off. Given witness descriptions, police mounted a search of the area but were unable to locate him.

Search for man who took hash from bundle washed up in Arenal

According to Which’s latest Consumer Insight Tracker, more than half of households – 55% – have made changes to cover essential spending as they continue to battle with increased financial pressures.

Around 2.1m Brits behind on essential bills and payments

Which? - According to Which’s latest Consumer Insight Tracker, more than half of households, 55%, have made changes to cover essential spending.

Around 2.1m Britons behind on essential bills and payments – Which?

The Office for National Statistics (ONS) said retail sales volumes fell by 0.9% in September against the previous month

UK retail sales slump as unseasonable warmth puts shoppers off buying autumn clothing

Tacoma Little Theatre has welcomed the fall season with Stephen King’s Misery.  Get a first look at photos below!

Photos: Get a First Look at MISERY at Tacoma Little Theatre

A warmer-than-usual September is expected to have led to depressed retail sales across the UK. Economists forecast the Office for National Statistics will report a 0.3% fall in retail sales in September on Friday, due to prolonged unexpected hot weather ...

Retail sales ‘hit by warm weather’ in September

Retail sales ‘hit by warm weather’ in September - Economists have forecast the Office for National Statistics will report a 0.3% fall due to unexpected hot weather during the month.

Economists have forecast the Office for National Statistics will report a 0.3% fall due to unexpected hot weather during the month.

UK inflation is expected to have decreased slightly last month due to a slowdown in food and clothing price increases. Official figures are predicted to reveal a decrease in Consumer Prices Index (CPI) inflation to 6.6% in September from 6.7% in August ...

Inflation set to cool again as food price rises ease

UK inflation is expected to have decreased slightly last month due to a slowdown in food and clothing price increases.Official figures are predicted to reveal a decrease in Consumer Prices Index (CPI) inflation to 6.

Deya

deya

UAE Researchers Create First AI That Pinpoints Objects in Videos, Down to the Pixel

New Method Could Unlock AI's Power to See and Describe Images with Unmatched Detail

DreamLLM: What We Can Conclude From This Comprehensive Framework?

NExT-GPT: Any-to-Any Multimodal LLM: Conclusion and References

* * Achievement unlocked: With more than ‘4 billion’ in total votes -- almost half the population of earth voted for startups! JUST KIDDING! Some companies had a to accumulating their votes. unique approach In 2021, decided to host to highlight the backbone of the future - tech startups! for the first time in forever (make sure you sing it in Anna’s voice), HackerNoon.com Startups of The Year More than 44,000 startups were nominated all across the globe, spanning over 4,553 cities. That’s.Just.Spiffy! To see who is a winner in each respective city, visit and pick a city. The name at the top of the list is the winner. (Compromised awards are under review.) startups.hackernoon.com The will announce the winners per region in their respective cities. Startups Winners Announcement Series We’d like to take a moment here and thank every single person who cast a vote and every single company, both nominated and not nominated yet this year. Furthermore, we thank you for existing and opening more opportunities for the wonders of the future to come. Even if you don’t win anything today, you’ve won something much more valuable - our hearts <3 Okay, sorry for the long introduction. Below are all the Startups Winners: Startups of the Year (SOTY) in Africa Johannesburg, South Africa With more than 900 votes cast, #SOTY calls.... MiBRAND Thank you for connecting businesses and consumers, great job! Lagos, Nigeria With 836 votes in the bag, let’s give it up for... Goodtalent Thanks for helping tech companies build great engineering teams. Cape Town, South Africa Want to learn more about finance and use that knowledge to make better financial decisions? Look no further. Introducing winner of Startups of The Year, Cape Town: Revix With 527 votes cast, it seems like a lot of people have more money in their pockets thanks to . Revix Mahé, Seychelles With around 515 votes cast, #SOTY summons... DAOLaunch Thank you for bridging the gap between investors and startups. Lusaka And the winner for #SOTY in Lusaka is... ComGrow Thanks to them, people here are getting closer to the reality of the digitization of village banking. Nairobi, Kenya With 252 votes cast, UTU came out top as the Startup of The Year. Thanks for building the trust infrastructure of the internet to help businesses and consumers engage and transact in a more reliable way. Yaoundé Accounting for more than 73% of total votes, CamPay won this award fair and square. Thank you for making sending and receiving payments easier between online merchants. Abuja, Nigeria A proud winner in the prestigious capital of Nigeria, Elesaro deserves all the plaudits it gets. is a suite crowdfunding protocol built on BSC, Reward, Equity, Lending & NFT Marketplace. Congratulations! Elesaro Dar Es Salaam, Tanzania Jamii Africa is the name that will always be engraved on the #SOTY trophy. As a mobile micro-health insurance, Jamii Africa has done a great job for the health sector in Tanzania. Kigali, Rwanda Another #SOTY medal in the bag for none other than …wait for it… BAG Innovation BAG Innovation applies Gamification to offer real-time access to experience-based learning for University students in East Africa. Everybody loves playing games, but using games to learn in school is on another level. Gauteng, South Africa And the award goes to... Planet42 Thanks for democratising mobility by enabling people ignored by banks to get access to a personal vehicle through a car subscription service. Once again, congratulations! Port Harcourt, Nigeria Cryptocurrency is the present and the future, as we’ve all come to know. FLITBASE makes these digital assets more accessible to everyone. #SOTY is proud to add you to our Hall of Fame. Cairo, Egypt With 18 votes won out of the 91 cast, it seems there was fierce competition. Congratulations to... SubsBase for winning Startups of The Year. SubsBase allows businesses to manage their subscribers, automate recurring billings, set pricing plans, and monitor subscription KPIs. Tunis, Tunisia Come one, come all! Witness the greatness of yet another wonderful startup in the #SOTY. Thank you... VYND for providing a fully integrated digital platform to help venue owners better serve their customers and grow their businesses. Stellenbosch, South Africa BankTech connects clients to a payment network where it provides a convenient interface to meet various payments needs. You’ve won #SOTY proudly! Pretoria #SOTY acknowledges... Grays Pharmaceuticals Thank you for all you do. Lekki, Nigeria Middletrust provides reliable escrow services in Africa. Congratulations on winning a #SOTY medal. Addis Ababa Thank you... Hello Solar for consistently accelerating sustainable energy solutions, especially for remote communities. With half of the votes cast, you beat the odds. Accra, Ghana UT Bank Ghana provides various commercial banking services to individuals and companies in Ghana. We cannot emphasize enough the importance of financial services to humankind. #SOTY is proud of all you do. Gîza, Egypt Big congratulations to... HOVO for winning this award. Your on-demand mobile app for ordering trucks is truly appreciated! Goma, Democratic Republic of Congo Congratulations for coming out on top of #SOTY, SillonBlue Technologies As an ICT Consulting and SaaS company, SillonBlue Technologies focuses on software and mobile app development, and data collection. Yaba, Nigeria With more than half of the votes cast, this nominee conquered undisputedly. Let’s give it up for... Kuda a no-fee full-service digital bank making banking affordable for all Africans on the planet. Morocco City Almes provides infrastructure and logistics services. Congratulations on earning a #SOTY medal. Dakar ShopMeAway provides families and businesses with new ways to shop around the world with reduced shipping costs. You deserve the plaudits you’ve received! Casablanca Sheaply provides crowd shipping and crowd shopping solutions. This reduces cost and improves efficiency on shipping and shopping. Congratulations on winning a #SOTY medal. Nguru, Nigeria Accounting for 88% of the votes, TradeBuza wins #SOTY almost flawlessly. TradeBuza is a tech-enabled commodities sourcing platform for commodities aggregators and out-grower schemes. Ikeja, Nigeria PropertyPro Nigeria is now pronounced Startup of the Year in Ikeja. Let’s admit, who doesn’t appreciate a platform built to search for real estate properties? Ibadan, Nigeria No Bank, No Problem! (That’s probably not their motto, by the way) Shield Finders is a fintech and financial service provider to the unbanked. Congratulation on winning most of the votes! 6 October City, Egypt Wuilt is a website builder where you can create your own Arabic website in under 5 minutes. Thank you for making this technology accessible to people. One #SOTY medal for you! Algiers The only nominee who outvoted all others, please give it up for... Yassir Thank you for providing convenient transport services using your app. Enugu In an area that usually suffers from inconsistent electricity, Greenage comes to the rescue with their green cape and spandex suit. Greenage Technologies is a Nigerian startup solving electricity problems in Nigeria using solar energy technology. Congrats, Greenage. Abeokuta, Nigeria A government organization creating opportunities for entrepreneurs and the unemployed - no one deserves the title Startup of The Year better than Ogun Innovate Congratulations! Ariana And the award goes to... E-Stores Factory a web platform that helps rent and manage e-shops for enterprises seeking to sell online. Thank you for making the (not a typo) more friendly! e-cosystem Benin City Health really is wealth. MOBicure provides mobile technology solutions for Africa's biggest health problems. Thank you for all you do. From #SOTY with love <3 Centurion GameZBoost offers white label gaming platforms, including online desktop solutions, fully responsive mobile and social platforms. Thank you for making the world a more fun place! Gaborone Augmenta Cyber Security provides services in cyber security, safe data storage, SCADA security, SAP Security, etc. Congratulations on winning 10/10 of the votes. You rock! Garki There can never be enough medical platforms in the world. Congratulations to... Wella Health an operator of a healthcare technology company intended to provide treatments and medicines for malaria. Thank you for all you do! Lilongwe Yellow is a digital platform enabling informal merchants in Africa. #SOTY welcomes you to its hall of fame. Make yourself at home ;) Luanda Access to finance is important for any company or sector to succeed. And no other company deserves the title Startup of The Year more than Deya the first Angolan and PALOP crowdfunding platform that aims to increase access to financing for social impact causes and entrepreneurs. . 10/10 will recommend Mombasa Life can be hard, and so a little humor helps ease these hardships. Congratulations to this political satire puppet show, The XYZ Show for winning #SOTY. Thanks for putting smiles on our faces. Port Louis Town Bet you’ve never heard of a Smartcup. Thanks to Intellicup that exists now, and therefore earns this startup a #SOTY badge.  Intellicup is a unique high-speed beverage dispensing solution with integrated smart technology. While this cup won’t be able to wake you up in the morning, you have to agree that it’s still pretty impressive. Rabat Atlan Space is a startup building AI to autonomously pilot any fixed-wing drone, to monitor and take measures over very large areas. You deserve the #SOTY medal. Réunion Imagine having the power to do ultrasound scans on your handheld device. OSCULT by Oscadi is a handheld ultrasound imaging device interfaced with the Apple iPad. Congratulations on making it into the #SOTY! Somerset West Winning the award fair and square is Sticitt a cashless payment platform that is also aimed at teaching learners at school the basics of finance. The future of the world would be in good hands when our children know how to adequately manage finances. Thank you for saving the world in advance. Thika Flowers make the world a beautiful place to live in. Wilmar Flowers exports summer flowers grown by small and medium-sized farmers from Kenya. Kudos for making the world beautiful, #SOTY is honored to have you in our Hall of Fame. Kampala, Uganda Yo-Waste Limited is an African startup that provides waste, recycling and “smart city” solutions to businesses and governments. Your #SOTY gold medal shines as much as you do <3 Harare, Zimbabwe The world is becoming more and more digital every day. Thanks to companies like Golix that provide services of digital currencies for people in Africa, the digitization of the world is becoming more robust. Freetown Easy Solar is a distribution company making energy, durable goods and financial services affordable and accessible through pay-as-you-go tech. Thumbs up! Kicukiro FOODLINE specializes in non-genetically-modified grocery farming and production lines. Winner Winner Chicken Dinner! #SOTY Goma, Democratic Republic of Congo Startups Nuru is a renewable and environmental company that specializes in the fields of storage micro-grids and solar energy. After racking up all the votes, #SOTY honors you. Abidjan, Côte d'Ivoire Digitech provides a fully integrated and cloud-based InsurTech platform to incumbent insurance companies. You are awesome! Durban Security can never really be enough. We’d like to honor Blue Security the title Startup of The Year, for specializing in the installation and monitoring of intruder detection systems, and in providing armed response services. Ikoyi And the Startup of The year goes to... Winich Farms an Agritech Startup leveraging tech to connect rural farmers to Finance & Market, thus, increasing productivity and cutting post-harvest loss. You rock! Lomé The last recipient of #SOTY in Africa calls for... Gozem a transport technology platform for Francophone West and Central Africa. Thank you for providing a platform for transportation. We are honored to have you! *Winners missing from this list will be added accordingly. More Information If you’re a winner, please answer ! If you’re a reader, participant, just a curious individual, consider answering one of our interviews . this interview here Read the FAQs about or . Startups here here To see all of the winners you may visit , pick any city and view the name that appears at the top. this site If you’re interested in partnering with us, we invite you to look through some of these or reach out to our business department. unique sponsorship options Perhaps creating your very own tickles your fancy or maybe you have a whole new idea for how we can partner! writing contest Thank you to all of our sponsors! We truly appreciate your contribution to the tech world and thus the future! The Startups Awards are sponsored by: , , , , , , , , , , Blockster Brex FreeCodeCamp LaunchDarkly Coil Udacity Udemy Vercel Asia Pacific Transgender Network Bybit Everscale Network This post was written and edited by Kien Dao and Joshua Omale. If you have any questions or concerns or would like to report a discrepancy please email ellen@hackernoon.com.

What are the Top Startups in Africa?

*Nota: Contactar a Omar Espejel (omar@tsc.ai) para cualquier observación. Cualquier error es responsabilidad del autor. *Nota: El autor considera que, si bien el escrito está en español y el objetivo es expandir el conocimiento del deep learning en este idioma, existen palabras insustituibles del inglés. Por lo tanto, este y siguientes escritos utilizarán indiscriminadamente palabras en inglés. Manipulación de tensores Se introduce . Usamos . Finalmente, aplicamos . Terminamos con una función que utilizaremos más adelante para aplanar imágenes. el tensor como la estructura de datos básica en la creación de deep learning pytorch como herramienta para la construcción de tensores capaces de correr en CUDA (GPU) manipulación de tensores para hacer broadcasting (transmisión, en español), agregar y quitar dimensiones, y hacer reshaping de nuestras estructuras ¿Dónde correr el código? En mi opinión, la mejor forma de aprender deep learning es . Esto es irrealizable, en terminos de tiempo, si se utiliza el poder de una CPU. Estaremos trabajando en Google Colab donde se nos asigna de manera gratuita por máximo 12 horas un GPU modelo Tesla K80. Solo necesitamos una cuenta de Google. escribir y escribir código Ingresar a y hacer click en . Ya podemos comenzar a escribir código python en la nube completamente gratis. Para acceder a un GPU dar click en - y en acelerador de hardware seleccionar GPU. Eso es todo, estamos listos para hacer deep learning. https://colab.research.google.com NUEVO BLOCK DE NOTAS DE PYTHON 3 Entorno de ejecución Cambiar tipo de entorno de ejecución El tensor En matemáticas estamos comunmente familiarizados con tres estructuras de datos: el escalar, el vector y la matriz. Los escalares corresponden a un único número, 2 por ejemplo; los vectores son una formación ordenada (arrays, en inglés) de datos en una única dimensión, por ejemplo (1,3,5,6,7,8); las matrices son una formación ordenada de dos dimensiones, por ejemplo ([1,2,3],[4,5,6]). Sin embargo, La siguiente imagen obtenida de nos permite visualizar estas estructuras de datos y la limitación al tratar de nombrar estructuras de más de dos dimensiones. ¿cómo nombramos a las estructuras de datos, como las imágenes, de más de 2 dimensiones? Leonardo Araujo Santos Los matemáticos tienen la tendencia de nombrar cada particularidad de una forma diferente, en deep learning usaremos una manera general de nombrar las estructuras de datos. . Para una explicación más detallada de lo que es un tensor ver este . Sin embargo, para los fines de este artículo, . No hablaré de escalares, vectores o matrices, sino de tensores de rango cero, uno y dos, respectivamente artículo pensemos en los tensores como una formación ordenada de N dimensiones (si se tienen conocimientos previos de lenguajes como R y Matlab entonces la sintaxis les parecerá conocida). Primero necesitamos importar Pyorch: . Ahora creemos un : Creemos tensores import torch tensor de rango cero t = torch.tensor( , dtype = torch.int32, device = ) 1 "cuda" . En este caso usamos enteros pero podemos incluir o para crear variables que almacenen números con puntos decimales. , en este caso utilizaremos CUDA para correr nuestros tensores en nuestra GPU. Si fueramos a correr nuestros modelos en un CPU entonces usamos . Vamos con tensores de rango 1 y 2: El argumento dtype selecciona el tipo de tensor torch.double torch.float64 En device seleccionamos el tipo de hardware a utilizar device = "cpu" Rango 1: t = torch.tensor([ , , ], dtype = torch.double64, device = ) 1.1 2.2 3.3 "cuda" Rango 2: t = torch.tensor([
    [ , , , ],
    [ , , , ],
    [ , , , ]
], dtype=torch.float32, device = ) 1 1 1 1 2 2 2 2 3 3 3 3 "cuda" , que en este caso nos regresa pues nuestro tensor tiene tres filas y cuatro columnas. Más adelante crearemos tensores de rango 3 que nos servirán como inputs para nuestros modelos. Para conocer la forma de un tensor utilizamos t.shape torch.Size([3, 4]) t Finalmente, . Usemos nuestro tensor de forma . podemos realizar operaciones entre tensores t torch.Size([3,4]) o . Suma: t + t t.sum(t) : . Multiplicación t*t : donde se define como: Multiplicación de matrices t.mm(s) s s = torch.tensor([
    [ , , ],
    [ , , ],
    [ , , ],
    [ , , ]
], dtype=torch.float32, device = ) 1 1 1 2 2 2 3 3 3 4 4 4 "cuda" Además, No ahondamos en el tema dado que no es tan relevante para iniciar con el deep learning, sin embargo, en la documentación de pytorch se encuentra todo lo necesario. pytorch nos permite hacer indexing de la misma manera que se hace con numpy. Broadcasting (transmisión, en español) El broadcasting es una . Esta técnica existe también en numpy. herramienta muy poderosa que nos permite trabajar mejor con tensores sin usar fors Cuando . . pytorch opera en dos tensores, compara sus formas en cuanto a elementos Comienza con las dimensiones finales y avanza Dos dimensiones son compatibles cuand n se transmite para que tenga el mismo tamaño. Esto significa que una dimensión con valor de 1 es compatible con cualquier otra dimensión, ya que se transmitirá. Por ejemplo, se puede operar en tensores de forma (1,3) y (5,1) ya que 1 es compatible con 3 y 5. o: (1) son iguales o (2) una de ellas es 1, en cuyo caso esta dimensió . Por ejemplo, si tienes un tensor 256 * 256 * 3 de valores RGB (red-green-blue), , y deseas escalar cada color de la imagen por un valor diferente, puedes multiplicar la imagen por un tensor de rango 1, , con 3 valores, es decir con forma (3). Los tamaños de los ejes finales de estos tensores, de acuerdo con las reglas de transmisión, los hace compatibles. Los tensores no necesitan tener el mismo número de dimensiones RGB_tensor t . creará un tensor en forma (1,1,3) donde el tercer eje incluye los valores que se aplicarán a cada uno de los colores en . te dará el tensor que puede multiplicar por la imagen de tamaño (256,256,3). Los ejes con valores 1 se transmitirán. Para hacer esto necesitamos compatibilizar nuestros tensores RGB_tensor y t . Para multiplicar cada dimensión del RGB_tensor de acuerdo con cada uno de los valores en t , necesitamos expandir a este último para tener 1s en el primer eje de t t = t[None, None,:] RGB_tensor t.expand_as(RGB_tensor) Estamos escribiendo código que se ejecuta tan rápido como C con esto. Además, el broadcasting nos da velocidad C sin necesidad de memoria adicional. . Toma un poco de tiempo acostumbrarse a la forma de codificación de broadcasting, pero, según Jeremy Howard en su clase de practical deep learning, vale la pena ya que reduce muchos errores provenientes de los bucles, y también es más cómodo una vez que nos acostumbramos De acuerdo a Jeremy Howard, el broadcasting podría ser el truco más útil para escribir código que sea lo suficientemente rápido. Agregar dimensiones o ejes En el texto se utilizarán de forma indiscriminada las palabras "dimensión" y "eje". . de modo que si el tensor originalmente tenía forma (3,750,750) después de la transformación, obtendremos una forma (1,3,750,750). . También funciona para numpy. Otro truco útil proviene de recordar que , por ejemplo, t.unsqueeze (-1) agrega un nuevo eje en la última dimensión. Hay dos formas diferentes de agregar una nueva dimensión a un Tensor de pytorch tensor.unsqueeze(dim = 0) , por ejemplo, agregará un nuevo eje en la dimensión 0 del tensor La segunda forma es la favorita de Jeremy Howard por su simplicidad. Lo mismo que antes se puede lograr escribiendo tensor[None,:,:, :) -1 siempre se refiere a la última dimensión del tensor Esto es relevante porque nos permite . . Por ejemplo, si tenemos un tensor, , con forma (3,3) y un tensor, , con forma , entonces creará un tensor horizontal de forma (1,3) que podemos expandir al tamaño de simplemente como que producirá un tensor en forma (3,3) que transmitirá los valores del vector horizontal para cada fila. elegir cómo transmitir nuestros tensores Considere que la transmisión se producirá automáticamente para la dimensión del tensor con forma 1 t s torch.Size([3]) s = torch.unsqueeze(s, dim = 0) t t.expand_as(t) Nota: . Squeeze hace lo contrario que unsqueeze(); elimina dimensiones Reshaping (remodelación, en español) de tensores Continuemos con nuestro tensor . : t Podemos conocer el rango y la forma del tensor con t.shape y len(t.shape) . t.numel() muestra el número de elementos en el tensor t.shape, len(t.shape), t.numel() que da como resultado . En , por lo tanto, tenemos 12 elementos acomodados en dos dimensiones. Los 12 elementos los podemos acomodar de diferentes en diferentes dimensiones, por ejemplo: (torch.Size([3, 4]), 2, 12) t t.reshape( , ), t.reshape( , ), t.reshape( , ); 1 12 6 2 12 1 Nota: . reshape() y view() hacen lo mismo “Veamos un caso de uso común para squeeze un tensor mediante la construcción de una función flatten (aplanar). […] Una operación de aplanar en un tensor da nueva forma al tensor para que tenga una forma que sea igual al número de elementos contenidos en el tensor. Esto es lo mismo que un tensor de rango 1°. […] Aplanar un tensor significa eliminar todas las dimensiones excepto una” - . Deeplizard En este texto, recreé la función de aplanar en el blog para tomar en cuenta los batches (lotes, en español) de datos. Indiquemos que nuestro tensor es un batch creando una dimensión adicional en la posición 0: t = t[ , :]
t.shape None Da . Ahora usemos la función para concatenar dos de forma que nuestro batch ahora tenga dos tensores. torch.Size([1, 3, 4]) cat t t = t[ , :]
t_batch = torch.cat((t, t), dim = ) None 0 retorna . Definimos entonces nuestra función para aplanar que toma en cuenta los batches. t_batch.shape torch.Size([2, 3, 4]) f = torch.reshape(t, (t.shape[ ], , )) f.squeeze() : def flatten (t) 0 1 -1 return Entonces , retorna , . t_batch.shape flatten(t_batch).shape (torch.Size([2, 3, 4]) torch.Size([2, 12])) “Dado que el argumento t puede ser cualquier tensor, pasamos -1 como segundo (tercer) argumento a la función reshape(). En pytorch, el -1 le dice a la función reshape() que descubra cuál debe ser el valor en función del número de elementos contenidos dentro del tensor. […] Veremos que se requieren operaciones de aplanamiento cuando se pasa un tensor de salida de una capa convolucional a una capa lineal.” . Deeplizard En el siguiente texto comenzaremos con los básicos del deep learning.

Manipulación de tensores en PyTorch. ¡El primer paso para el deep learning!

Opening of the Crypto Valley Conference and introduction by Oliver Bussmann, President of , and one of the top blockchain influencers in the world. Crypto Valley Association Switzerland is Europe’s answer to the Silicone Valley, Malta, and Singapore. The country offers one of the key hubs for crypto technology development. The majority of infrastructure is located in Crypto Valley within a unique district of Zug. It is the mothership for the ecosystem. A global center where emerging , blockchain, and other distributed ledger technologies rise. There, they develop in a highly supportive and safe environment. blockchain cryptocurrency Getting to Switzerland was a collaboration of our interest with the support and eagerness of our clients. Anton Dyshkant, the business development director at Crypto Gang, traveled to Crypto Valley to meet and cooperate with our existing clients there. He also had a chance to participate at the International . Below is a summary of highlights Crypto Gang is most excited to share. Crypto Valley Conference on Blockchain Technology The Event The 2018 International Crypto Valley Conference was organized by , and took place on June 20–22 of 2018, in Zug. It brought together attendees from America and Europe to discuss state of the art advances within the Blockchain Technology ecosystem. Crypto Valley Opening of the Crypto Valley conference. We took part in a truly unique experience. The event only hosted 800 attendees all of whom were industry professionals. These individuals are fully committed to developing the blockchain sphere and are active contributors to its progress. At the pre-event of Crypto Valley conference arranged by . With Olga Feldmeier — CEO of Smart Valor, a former executive of UBS, Barclays Investment Bank, The Boston Consulting Group (BSG), and a well-known influencer in crypto world. Token Foundry Among the speakers were representatives from such organizations as: Ripple, Microsoft, Cornell University, Swiss Government, UC Berkeley, IBM Research Zurich, Cisco, Eternity, and many others. Everyone was focused on quality networking opportunities. The event hummed with innovation and exuded creativity sparked by international collaboration. Listed below are a few of the most notable attendees: Oliver Bussmann — the advisor for top crypto projects ( , , ) IOTA Ripple Shapeshift Prof. Dr. Emin Gun Sirer of Cornell University Stefan Thomas — the inventor of the Interledger Protocol and CTO of Ripple Dolfi Mueller — the mayor of Zug who vigorously supports the blockchain throughout Switzerland Daniel Haudenschild — head of Swisscom Blockchain AG Richard Olsen — CEO of Lykke With Richard Olsen, founder of foundation. Crypto Gang is partnering with Swiss-based Lykke providing branding services to its clients. Apart from ICO services, Lykke is a crypto exchange and a digital wallet for crypto currencies and one of the leading service providers in Switzerland. Lykke The conference covered a wide range of topics. It touched on everything from the future of token economy, fintech, and IoT to smart contract security. Some of the other issues discussed included: identity management, shared data, micropayments, and legal aspects of cryptocurrency infrastructure. It’s impossible not to comment on the impeccable job the Crypto Valley Association did on planning and carrying out this event. They had never hosted a conference of such size and proportions before, and yet everything from the guest list to the numerous satellite subunits were organized perfectly. The satellite element was especially note-worthy as it allowed industry professionals to closely speak with some of the startups and less known companies within the industry. Together with a great pal, Shay Gammer, co-founder of Montfort Family Office and COO at based in Geneva. Fortune Family Office Group The Valley of Crypto Opportunities Crypto Gang came to Zug with intent to visit the cradle of crypto projects in Switzerland.Here, crypto is supported at a state level. Government officials make it a priority to create the best business climate for crypto currency development. Zug is a pretty small and picturesque place with the population of 28,000. Despite its small size, thanks to favorable regulatory framework and taxes, this region in Switzerland is considered to be one of the most interesting places to launch tech and blockchain projects. Not to much surprise, it has already attracted some of the world’s most renowned crypto companies and organisations such as: Ethereum, Monetas, Bitcoin Suisse, Xapo, ShapeShift, ConsenSys, and Tezos. With Oliver Busman, President of the Crypto Valley Association, CEO and Founder of Bussmann Advisory consultancy firm, in his office. We were amazed to discover that almost every resident knew about blockchain, crypto, and the conference. Almost everyone we met had opinions and ideas to share regarding their experience with crypto proving just how significant this technology has become for Zug as well as Switzerland as a whole. As crypto evangelists we strongly hope that very soon other countries will reach this level of crypto interest and awareness. We look forward to the time when crypto communication will enter the mainstream culture of every town out there. Networking and Communication At the booth of Bitcoin Suisse together with Nicolai Oster, Head of ICO and Partner. Discussing collaborations between our companies on ICOs and blockchain projects. For everyone in the crypto community such conferences are a good chance to connect with other professionals and enthusiasts within the industry. It is especially valuable for crypto startups and ICOs as they aim to test out elevator speech and product ideas, gathering feedback prior to launch. It also lets companies meet their investors and future supporters. At Crypto Gang we’re constantly encouraging our clients to travel to such events and network. One of the satellite events of the conference — “ICO investments”, led by Heinrich Zetlmayer, who is the founder and General Partner of the BVV, and former Vice President of IBM. Crypto Gang is partnering with BVV, Heinrich and their great team. It was a great honor to be one of the few companies from Ukraine to be part of the international dialogue about main points in the modern niches of start-ups. But what’s more important, we had an amazing opportunity to meet some of our partners and discuss our progress. We also fulfilled our goal of connecting and gaining support from some of the key market players interested in further cooperative partnerships. We are excited to share more details about all these points in upcoming releases so please check back for updates. Together with Dr. Mattia L. Rattaggi, a great person and a blockchain industry professional, who is a Chair Regulatory Policy at Crypto Valley Association and Founder of the named after him. blockchain Advisory and Consulting firm We also had a pleasure to meet some of our Swiss-based clients and discuss ongoing projects’ activities. We met with Robert Rogenmoser of . They are based in Zurich and produce hardware and software products that already protect the Swiss Banking System by securing €100 Billion in daily transactions and serving top Swiss financial institutions. We had a chance to discuss their upcoming ICO/ITO project and our progress on designing their ICO/ITO website, whitepaper, and other documents. Securosys With Robert Rogenmoser. Together with one of the conference attendees, Roland Kask, a marketing professional from Estonia. We stopped by the office of our good partner agency Lykke and their newly opened location in Zug at Alpenstrasse 9 where we met with Michael Guzik, Victor Solomon and others. We also met Heinrich Zetlmayer and Luigi Bruno along with the team of . BVV is a new accelerator and venture firm that focuses on incubating, developing, and investing in blockchain-enabled businesses. We are currently cooperating on projects together, and were eager to discuss our progress. Blockchain Valley Ventures With Deyana Nedeva, Head of Investor Relations at BVV. Overall, it was a great event! At Crypto Gang we strongly encourage everyone to regularly visit collaborative events in their niche to share, connect, and develop. We hope to see you somewhere in the world soon! Want to consult on how to make your ICO design great, see our portfolio or chat? Email me on . anton@cryptogang.agency

Crypto Gang visits Crypto Valley — the Swiss heart of European Blockchain

See the Stunning Details This AI Preserves in Relit Photos

This is Part 9 of a 12-part series based on the research paper “Human-Machine Social Systems.” Use the table of links below to navigate to the next part. Table of Links Abstract and Introduction Human-machine interactions Collective outcomes Box 1: Competition in high-frequency trading markets Box 2: Contagion on Twitter Box 3: Cooperation and coordination on Wikipedia Box 4: Cooperation and contagion on Reddit Discussion Implications for research Implications for design Implications for policy Conclusion, Acknowledgments, References, and Competing interests Implications for Research Existing research is often biased towards engineering and optimization, lacking deeper insights from a social science perspective. The time for a new sociology of humans and machines is critical, before AI becomes more sophisticated: generative AI exhibits emergent behavior that itself requires explanation[220,221], complicating the understanding of system dynamics. Researchers would benefit from an agent-based modeling framework that outlines distinctions between human and bot agents: utility function, optimization ability, access to information, learning, innovation/creativity, accuracy, etc. The framework could borrow concepts from other two-agent systems, such as predator–prey, principal–agent, and common pool resource models. Controlled experiments should explicitly compare human-machine, human-only and machine-only networks, and known bots against covert bots. Experiments could manipulate participants’ perceptions of algorithms’ technical specifications, agenthood[222], emotional capability, and biases. Field interventions in online communities with endemic bot populations present another promising direction. Existing examples include social bots that gain influence by engaging human users [223,224,225,226], trading bots that manipulate prices in cryptocurrency markets [227], political bots that promote opposing political views to decrease polarization[228], and “drifters” to measure platform bias [94]. Expanding on the cases reported here, we need observational research on additional human-machine communities and contexts such as traffic systems with human-driven and driverless vehicles, online multiplayer games comprising human players, non-player characters, and cheating code, and dating markets with AI-driven chatbots [229]. Finally, research with artificial agents introduces ethical problems demanding careful elaboration and mitigation. Research protocols should minimize interventions [230], possibly deploying covert bots only where they already exist, ensuring their actions are not unusual or harmful [93]. Even then, bots may still face opposition from users due to privacy concerns [223]. Do people perceive certain bots as inherently deceptive? Could knowledge of the bot owner and algorithm mitigate this perception? Authors:
(1) Milena Tsvetkova, Department of Methodology, London School of Economics and Political Science, London, United Kingdom;
(2) Taha Yasseri, School of Sociology, University College Dublin, Dublin, Ireland and Geary Institute for Public Policy, University College Dublin, Dublin, Ireland;
(3) Niccolo Pescetelli, Collective Intelligence Lab, New Jersey Institute of Technology, Newark, New Jersey, USA;
(4) Tobias Werner, Center for Humans and Machines, Max Planck Institute for Human Development, Berlin, Germany. This paper is available on arxiv under CC BY 4.0 DEED license. This is Part 9 of a 12-part series based on the research paper “Human-Machine Social Systems.” Use the table of links below to navigate to the next part. This is Part 9 of a 12-part series based on the research paper “Human-Machine Social Systems.” Use the table of links below to navigate to the next part. “ Human-Machine Social Systems .” Human-Machine Social Systems Table of Links Abstract and Introduction Abstract and Introduction Human-machine interactions Human-machine interactions Collective outcomes Collective outcomes Box 1: Competition in high-frequency trading markets Box 1: Competition in high-frequency trading markets Box 2: Contagion on Twitter Box 2: Contagion on Twitter Box 3: Cooperation and coordination on Wikipedia Box 3: Cooperation and coordination on Wikipedia Box 4: Cooperation and contagion on Reddit Box 4: Cooperation and contagion on Reddit Discussion Discussion Implications for research Implications for research Implications for design Implications for design Implications for policy Implications for policy Conclusion, Acknowledgments, References, and Competing interests Conclusion, Acknowledgments, References, and Competing interests Implications for Research Existing research is often biased towards engineering and optimization, lacking deeper insights from a social science perspective. The time for a new sociology of humans and machines is critical, before AI becomes more sophisticated: generative AI exhibits emergent behavior that itself requires explanation[220,221], complicating the understanding of system dynamics. Researchers would benefit from an agent-based modeling framework that outlines distinctions between human and bot agents: utility function, optimization ability, access to information, learning, innovation/creativity, accuracy, etc. The framework could borrow concepts from other two-agent systems, such as predator–prey, principal–agent, and common pool resource models. Controlled experiments should explicitly compare human-machine, human-only and machine-only networks, and known bots against covert bots. Experiments could manipulate participants’ perceptions of algorithms’ technical specifications, agenthood[222], emotional capability, and biases. Field interventions in online communities with endemic bot populations present another promising direction. Existing examples include social bots that gain influence by engaging human users [223,224,225,226], trading bots that manipulate prices in cryptocurrency markets [227], political bots that promote opposing political views to decrease polarization[228], and “drifters” to measure platform bias [94]. Expanding on the cases reported here, we need observational research on additional human-machine communities and contexts such as traffic systems with human-driven and driverless vehicles, online multiplayer games comprising human players, non-player characters, and cheating code, and dating markets with AI-driven chatbots [229]. Finally, research with artificial agents introduces ethical problems demanding careful elaboration and mitigation. Research protocols should minimize interventions [230], possibly deploying covert bots only where they already exist, ensuring their actions are not unusual or harmful [93]. Even then, bots may still face opposition from users due to privacy concerns [223]. Do people perceive certain bots as inherently deceptive? Could knowledge of the bot owner and algorithm mitigate this perception? Authors: (1) Milena Tsvetkova, Department of Methodology, London School of Economics and Political Science, London, United Kingdom; (2) Taha Yasseri, School of Sociology, University College Dublin, Dublin, Ireland and Geary Institute for Public Policy, University College Dublin, Dublin, Ireland; (3) Niccolo Pescetelli, Collective Intelligence Lab, New Jersey Institute of Technology, Newark, New Jersey, USA; (4) Tobias Werner, Center for Humans and Machines, Max Planck Institute for Human Development, Berlin, Germany. Authors: Authors: (1) Milena Tsvetkova, Department of Methodology, London School of Economics and Political Science, London, United Kingdom; (2) Taha Yasseri, School of Sociology, University College Dublin, Dublin, Ireland and Geary Institute for Public Policy, University College Dublin, Dublin, Ireland; (3) Niccolo Pescetelli, Collective Intelligence Lab, New Jersey Institute of Technology, Newark, New Jersey, USA; (4) Tobias Werner, Center for Humans and Machines, Max Planck Institute for Human Development, Berlin, Germany. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

AI's Complexity Demands Urgent Study of Trust, Bias, and Behaviors in Diverse Contexts

Authors:
(1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors;
(2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors;
(3) Sahal Shaji, Mohamed bin Zayed University of AI;
(4) Abdelrahman Shaker, Mohamed bin Zayed University of AI;
(5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University;
(6) Hisham Cholakkal, Mohamed bin Zayed University of AI;
(7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University;
(8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University;
(9) Ming-Hsuan Yang, University of California - Merced and Google Research;
(10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 1 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Table of Links Abstract and 1 Introduction
2. Related Work
3. Method
4. Data Annotation Pipeline
5. Experiments
6. Conclusion and References Supplementary Material (Part 1) A. Additional Implementation Details
B. Additional Downstream Tasks
C. Additional Qualitative Results Supplementary Material (Part 2) D. Dataset Visualization
E. Limitations and Future Work
F. Ethics and Societal Impact Abstract Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the conversations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we introduce a comprehensive evaluation protocol with our curated grounded conversations. Our proposed GCG task requires densely grounded concepts in natural scenes at a large-scale. To this end, we propose a densely annotated Grounding-anything Dataset (GranD) using our proposed automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations. 1. Introduction Fueled by the generative AI wave, Large Multimodal Models (LMMs) have emerged as a pivotal advancement, bridging the gap between vision and language tasks [2]. Initial efforts like [6, 8, 22, 29, 52, 61] demonstrate effective textual responses based on input images. Although these models are sophisticated, they cannot still ground their responses in the visual context. Such grounding is crucial for advanced applications like detailed visual understanding, interactive embodied agents, and localized content manipulation. Recent efforts have started to address this limitation by enabling models to process user-defined regions specified via bounding boxes [5, 31, 35, 36, 57]. A few recent works have explored grounded text response generation [5, 21, 35, 59] but do not provide detailed pixel-level groundings. Parallel to these, efforts have been made in the referring segmentation literature to ground textual descriptions in natural images [21]. However, they are limited to grounding a single object and cannot engage in natural, coherent conversations, thereby restricting their practical applicability in interactive tasks that demand a deep understanding of both visual and textual content. To address these limitations of existing works, we introduce Grounding LMM (GLaMM), that simultaneously provides in-depth region understanding, pixel-level groundings, and conversational abilities through an end-to-end training approach (see Fig. 1 and Tab. 1). To address the lack of benchmarks for visually grounded conversations, we introduce the novel task of Grounded Conversation Generation (GCG). The GCG task aims to produce natural language responses interleaved with object segmentation masks. This challenging task unifies several existing tasks in computer vision that are typically treated in isolation, i.e., referring expression segmentation, image and region-level captioning, phrase grounding, and vision-language conversations. Thereby, our unified model and proposed pretraining dataset can effectively transfer to several downstream tasks (referring expression segmentation, region-level captioning, image captioning, and conversational-style QA). We present GLaMM as the first model specifically designed for this challenging task. Unlike prior works, GLaMM can work with both textual and visual prompts and can generate visually grounded outputs, thus offering a versatile user experience. Detailed region-level understanding requires the laborious process of collecting large-scale annotations for image regions. We propose an automated pipeline to annotate the large-scale Grounding-anything Dataset (GranD) to alleviate the manual labeling effort. Leveraging the automated pipeline with dedicated verification steps, GranD comprises 7.5M unique concepts anchored in 810M regions, each with a segmentation mask. Using state-ofthe-art vision and language models, the dataset annotates SAM [18] images through a multi-level hierarchical scheme that enhances annotation quality. With 11M images, 84M referring expressions, and 33M grounded captions, GranD sets a new benchmark in comprehensiveness. In addition to the automatically generated dataset for the GCG, we provide the first high-quality dataset for grounded conversations obtained by revamping the existing manually annotated datasets [16, 37, 49] for GCG using GPT-4 [34] incontext learning. We refer to the high-quality dataset as GranDf , denoting its suitability for fine-tuning. Our work has three main contributions: • We present GLaMM, the first model capable of generating natural language responses seamlessly integrated with object segmentation masks. Unlike existing models, GLaMM accommodates textual and visual prompts, facilitating enhanced multimodal user interaction. • Recognizing the lack of standardized benchmarks for visually grounded conversations, we propose the new Grounded Conversation Generation (GCG) task. We also introduce a comprehensive evaluation protocol to measure the efficacy of models for GCG that unifies multiple isolated tasks, filling a significant gap in the literature. • To facilitate model training and evaluation, we create Grounding-anything Dataset (GranD), a large-scale densely annotated dataset. Developed using an automatic annotation pipeline and verification criteria, it encompasses 7.5M unique concepts grounded in 810M regions. Additionally, we propose GranDf , a high-quality dataset explicitly designed for the GCG task finetuning, by repurposing existing open-source datasets. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors; (2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors; (3) Sahal Shaji, Mohamed bin Zayed University of AI; (4) Abdelrahman Shaker, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Hisham Cholakkal, Mohamed bin Zayed University of AI; (7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University; (8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University; (9) Ming-Hsuan Yang, University of California - Merced and Google Research; (10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Authors: Authors: (1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors; (2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors; (3) Sahal Shaji, Mohamed bin Zayed University of AI; (4) Abdelrahman Shaker, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Hisham Cholakkal, Mohamed bin Zayed University of AI; (7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University; (8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University; (9) Ming-Hsuan Yang, University of California - Merced and Google Research; (10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 1 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 1 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 1 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 1 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Table of Links Abstract and 1 Introduction 2. Related Work 3. Method 4. Data Annotation Pipeline 5. Experiments 6. Conclusion and References Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Method 3. Method 4. Data Annotation Pipeline 4. Data Annotation Pipeline 5. Experiments 5. Experiments 6. Conclusion and References 6. Conclusion and References Supplementary Material (Part 1) Supplementary Material (Part 1) A. Additional Implementation Details B. Additional Downstream Tasks C. Additional Qualitative Results A. Additional Implementation Details A. Additional Implementation Details B. Additional Downstream Tasks B. Additional Downstream Tasks C. Additional Qualitative Results C. Additional Qualitative Results Supplementary Material (Part 2) Supplementary Material (Part 2) D. Dataset Visualization E. Limitations and Future Work F. Ethics and Societal Impact D. Dataset Visualization D. Dataset Visualization E. Limitations and Future Work E. Limitations and Future Work F. Ethics and Societal Impact F. Ethics and Societal Impact Abstract Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the conversations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we introduce a comprehensive evaluation protocol with our curated grounded conversations. Our proposed GCG task requires densely grounded concepts in natural scenes at a large-scale. To this end, we propose a densely annotated Grounding-anything Dataset (GranD) using our proposed automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations. Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the conversations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we introduce a comprehensive evaluation protocol with our curated grounded conversations. Our proposed GCG task requires densely grounded concepts in natural scenes at a large-scale. To this end, we propose a densely annotated Grounding-anything Dataset (GranD) using our proposed automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations. 1. Introduction Fueled by the generative AI wave, Large Multimodal Models (LMMs) have emerged as a pivotal advancement, bridging the gap between vision and language tasks [2]. Initial efforts like [6, 8, 22, 29, 52, 61] demonstrate effective textual responses based on input images. Although these models are sophisticated, they cannot still ground their responses in the visual context. Such grounding is crucial for advanced applications like detailed visual understanding, interactive embodied agents, and localized content manipulation. Recent efforts have started to address this limitation by enabling models to process user-defined regions specified via bounding boxes [5, 31, 35, 36, 57]. A few recent works have explored grounded text response generation [5, 21, 35, 59] but do not provide detailed pixel-level groundings. Parallel to these, efforts have been made in the referring segmentation literature to ground textual descriptions in natural images [21]. However, they are limited to grounding a single object and cannot engage in natural, coherent conversations, thereby restricting their practical applicability in interactive tasks that demand a deep understanding of both visual and textual content. To address these limitations of existing works, we introduce Grounding LMM (GLaMM), that simultaneously provides in-depth region understanding, pixel-level groundings, and conversational abilities through an end-to-end training approach (see Fig. 1 and Tab. 1). To address the lack of benchmarks for visually grounded conversations, we introduce the novel task of Grounded Conversation Generation (GCG). The GCG task aims to produce natural language responses interleaved with object segmentation masks. This challenging task unifies several existing tasks in computer vision that are typically treated in isolation, i.e., referring expression segmentation, image and region-level captioning, phrase grounding, and vision-language conversations. Thereby, our unified model and proposed pretraining dataset can effectively transfer to several downstream tasks (referring expression segmentation, region-level captioning, image captioning, and conversational-style QA). We present GLaMM as the first model specifically designed for this challenging task. Unlike prior works, GLaMM can work with both textual and visual prompts and can generate visually grounded outputs, thus offering a versatile user experience. Detailed region-level understanding requires the laborious process of collecting large-scale annotations for image regions. We propose an automated pipeline to annotate the large-scale Grounding-anything Dataset (GranD) to alleviate the manual labeling effort. Leveraging the automated pipeline with dedicated verification steps, GranD comprises 7.5M unique concepts anchored in 810M regions, each with a segmentation mask. Using state-ofthe-art vision and language models, the dataset annotates SAM [18] images through a multi-level hierarchical scheme that enhances annotation quality. With 11M images, 84M referring expressions, and 33M grounded captions, GranD sets a new benchmark in comprehensiveness. In addition to the automatically generated dataset for the GCG, we provide the first high-quality dataset for grounded conversations obtained by revamping the existing manually annotated datasets [16, 37, 49] for GCG using GPT-4 [34] incontext learning. We refer to the high-quality dataset as GranDf , denoting its suitability for fine-tuning. Our work has three main contributions: • We present GLaMM, the first model capable of generating natural language responses seamlessly integrated with object segmentation masks. Unlike existing models, GLaMM accommodates textual and visual prompts, facilitating enhanced multimodal user interaction. • Recognizing the lack of standardized benchmarks for visually grounded conversations, we propose the new Grounded Conversation Generation (GCG) task. We also introduce a comprehensive evaluation protocol to measure the efficacy of models for GCG that unifies multiple isolated tasks, filling a significant gap in the literature. • To facilitate model training and evaluation, we create Grounding-anything Dataset (GranD), a large-scale densely annotated dataset. Developed using an automatic annotation pipeline and verification criteria, it encompasses 7.5M unique concepts grounded in 810M regions. Additionally, we propose GranDf , a high-quality dataset explicitly designed for the GCG task finetuning, by repurposing existing open-source datasets. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Stories, News & Mentions about DEYA

Deya’s Stories on HackerNoon

Deya's News & Mentions