texts: User activity representation for Prediction/ML

texts:
I want to predict future user activities (e.g., account cancellation) but I don't know how I'm supposed to represent the data.

The raw data is a sequence of all activities by all users:

2015-01-01T04:04|user1|created account
2015-01-01T05:04|user2|created account
2015-01-01T06:04|user1|changed plan
2015-01-01T07:04|user2|changed plan
2015-01-01T08:04|user1|cancelled submission

In the simplest prediction I'd like to label a user as 0=Cancelled and 1=Not Cancelled and subsequently would train a classifier for this label.

The question is how would I prepare the data for a machine learning algorithm (e.g. SVM or logistic regression)?. I now have multiple rows per user which are ordered by the time they happened, but I'm thinking of having only one row per user. How would I represent the activities? Furthermore, the number of activities per user is not limited (1 to n where n would hardly ever exceed 10), and the the number of features per activity can be large (up to 100 features).

I'd really appreciate it if anyone could help me out here :-)

Thx


texts: PCA-like analysis for dataset that has both categorical and continuous variables

texts: I have a dataset containing a categorical variable and multiple continuous variables. The categorical variables are coded as discrete integers, whereas the continuous variables are just a range of floats. I believe that the variance in my dataset can be almost entirely described by the single categorical variable and one of the many continuous variables. To justify this, I would be interested in using PCA, but I'm not sure the best approach to use when I am considering categorical data. Any suggestions?


texts: Regarding sklearn adjusted_rand_score in k means clustering

texts:
A dataset is given consisting of target class as categorical.K means was applied to it for clustering the data and got the corresponding cluster labels.   But how to put the values in adjusted Rand score function as it is taking the true values and predicted values and the true values are in categorical format.  I tried converting the target class true data to numeric by using LabelEncoder but when passed to adjusted Rand score it is coming low.
  Please suggest a better and correct approach


texts: Use regression instead of classification for hard labeled ranking datasets

texts:
Let's imagine I have a dataset of movie reviews with annotated sentiment:

-1 means negative
 0 means neutral
+1 means positive

I see a lot of people trying to do classification to try to answer those types of problems, but shouldn't regression be used instead? To me using regression would allow the system to model that there is a transition between labels, e.g. 0 is in between. Any thoughts on this?


texts: How does Keras calculate accuracy from the classwise probabilities? Say, for example we have 100 samples in the test set which can belong to one of two classes. We also have a list of the classwise probabilites. What threshold does Keras use to assign a sample to either of the two classes?


texts: Multiple confusion matrix for multiple training instances. Which one to take?

texts: I am using Matlab Neural Network toolbox for a classification problem. Now  considering a single set of data, if the inbuilt neural network is trained and classified with same data multiple number of time, different accuracy and different confusion matrix is obtained. Now which result should I take? Should I take all the vales obtained in all the training instances and average them fix on one particular result?


texts: Theoretical results on restricting the size of a neural network?

texts:
I am trying to create an experiment to see if a Neural Network (NN) can learn binary search algorithm or not.
My input matrix $X_{n\times 16}$ is a group of $n$ sorted lists, each of size $16$. I also have that $X_{i,j}\in \{1, 2, ..., 256\}$. I am interested in building a neural network that is able to search a list of size $16$, and find if it has the number $42$ or not.
In my current NN setup, the input is a $16x1$ sorted vector, and the output is a binary $0/1$ depending on whether the $16$ numbers have $42$ or not. My current NN is quite big (several wide layers), but is able to detect whether $42$ exists in the test input with $\approx 98\%$ accuracy ($42$ exists in half the input lists).
However, I realise that the NN might be just learning to look at every single element, comparing it to $42$, essentially performing $16$ comparisons.
I am now interested in shrinking the size of the NN to see if the optimiser can "find" the binary search approach. However, shrinking the NN too much might just lead it to not being able to learn anything.
So my question is, are there any theoretical results/bounds on the size of the NN (in terms of layers and width of layers) in comparison with the function it is trying to learn? Specifically, I am trying to force the NN to implicitly learn a procedure that takes precisely $\log_2(16)$ comparisons, as opposed to $16$ comparisons.


texts: I use logistic regression. We know that it is a supervised method and needs calculated feature values both in training and test data. There are six features. Although the functions produce these features’ values are different and their maximum value can be 1, there are four features (both in training and test data) that have very low values. e.g. they range between 0 and 0.1 and are never 1, even more than 0.1!!!. Thus these features’ values are very close to each other. Other features are distributed normally (they range between 0 and 0.9). So the difference between these two kinds of features is high, I think this causes trouble in learning process for logistic regression. Am I right?! Does it need any transforming/normalizing these features? Any help would be highly appreciated.


texts: Theoretical background for model arhitecture choosing

texts:
Suppose I have a dataset $X$ and two different binary labels $y_{1}$ and $y_{2}$. The classes are very imbalanced - 3% of true in $y_{1}$  and 2% in $y_{2}$. Moreover, there are no pairs of (0,1), so, if the  $y_{1} = 0$, then  $y_{2}=0$.

I have to choose between:

Associating the labels with each other, so it's a 3-class classification problem
Build a model for  $y_{1}$ scoring, then joining to $X$ and making scoring of  $y_{2}$, using $X$ and  $y_{1}$.

So the first idea is formalised as :

$$ y_{f} = y_{1} \bullet y_{2}; \,\,\,\, y_{f} = \{(1,1),(0,0),(1,0) \}; \,\,\, y_{f} = \Phi_{f}(X)$$

And the second road my be formalised like this:
$$ y_{1} = \Phi_{1}(X); \,\,\, \, \, X=X\cup y_{1};  \,\,\,\, y_{2}=\Phi_{2}(X) $$
Taking into account all condition, what is the possible theoretical background of projecting the best algorithm?  I'am trying to give estimates to an error, which is :bias + variance + noise. E.g.:
For both: $\Phi_{f}$ and $\Phi_{2}$, which are RandomForests, bias and noise will be the same, variance will be higher for  $\Phi_{f}$(just intuition till now). Can somebody point me the right way?


texts: Continuously training one model with different dimensions each time

texts:
I'm trying to solve a problem where I need to train one model with N dimensions and again train on top of that model with M dimensions. How can I achieve it?

To give you guys some context, I have 1 bluetooth beacon talking to 5 bluetooth readers that give out some value. Based on the value given out by the reader, I'm predicting the location of the beacon. My training data has 5 dimensions because there are values from 5 readers.

Now I have another scenario where I have 10 readers instead of 5. How do I train on top of my first model with 10 readers this time? Is it even possible?

The number of readers varies depending on the size of the room. So there's no way I can tell that there will always be a fixed number of readers (that is, fixed number of dimensions).

How do I go about solving this problem? Any help is appreciated!


texts: I have a pdf file (admission application). I want to read/search the pdf and extract terms with similar meaning and then convert this data into a DataFrame to save as a xlsm file. HELP!


texts: Can we combine multiple K-Means Models as a single model?

texts:
I have a NLP problem statement where I use a Word2Vec embedding pre-trained model to convert key text to vectors and then on a set of terms run k-means clustering to get a final model for certain k
For various sets of terms, I would develop a different model, which I would store to disk.
My question is, in case there is a new term, which I wish to classify as to which cluster should it point to from all the models can I follow the following approach?

Load all models to memory and get their cluster centers.
get the vector of the new term based on the same pre-trained model as before.
get distance from each cluster center to the new vector and whichever is nearest can be considered as the winning cluster

I would like to know what could be the possible drawbacks of such an approach.
My assumption is that since the vector space is same as defined by the pre-trained model, therefore the cluster centers would be in the same space.


texts: Classification and variable selection with a single known class

texts:
I am looking for help and suggestions on how to approach a classification and feature selection problem making use of a single define class and several unknown ones. Using an example, for those than know the iris dataset, imagine if we knew that there was at least one species, versicolor, and now we wanted to find out which other species could the dataset contain and how they would differ from versicolor. My data is a little more complicated than this but this is a fairly good simplification.
Clustering and feature reduction alternatives are also welcome. Solutions or pointers to R libraries would be perfect.
The problem itself:
I have a few separate experimental datasets with about 30 measurement variables, all numeric, and hundreds of observations (let's call them subjects). About 20 of those observations belong to a control class. The goals are:

identify which subjects are distinct from the control class;
determine which variables (features) are important for this distinction, or make these observation different;

The complications are:

in any given given experiment there could more than one non-control class
I expect only a small subset of the subjects to be different from the control class and
what makes separates them from the control may or may not be the same.

Notes on the data

Measurement for non-control classes were taken in triplicate
Some measurement variables are highly correlated.

What I have done so far and first approach:

PCA for feature reduction and cluster visualization
k-means clustering to identify classes.

This has been sub-optimal because it will tend to result in too many clusters because too few subjects are clearly different from the controls - outliers so to speak.


texts: plotting a decision tree based on gridsearchcv

texts:
i was trying to plot the decision tree which is formed with GridSearchCV, but its giving me an Attribute error.

AttributeError: 'GridSearchCV' object has no attribute 'n_features_'

However if i try to plot a normal decision tree without GridSearchCv, then it successfully prints.

code [decision tree without gridsearchcv]

# dtc_entropy : decison tree classifier based on entropy/information Gain
#plotting : decision tree on information/entropy  based

from sklearn.tree import export_graphviz
import graphviz

feature_names = x.columns

dot_data = export_graphviz(dtc_entropy, out_file=None, filled=True, rounded=True,
                                feature_names=feature_names,
                                class_names=['0','1','2'])
graph = graphviz.Source(dot_data)
graph                           ### --------------> WORKS

code [decision tree with gridsearchcv]

#plotting : decision tree with GRIDSEARCHCV (dtc_gscv)  on information/entropy  based
from sklearn.tree import export_graphviz
import graphviz

feature_names = x.columns

dot_data = export_graphviz(dtc_gscv, out_file=None, filled=True, rounded=True,
                                feature_names=feature_names,
                                class_names=['0','1','2'])
graph = graphviz.Source(dot_data)
graph                            ##### ------------> ERROR

Error

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-201-603524707f02> in <module>()
      6 dot_data = export_graphviz(dtc_gscv, out_file=None, filled=True, rounded=True,
      7                                 feature_names=feature_names,
----> 8                                 class_names=['0','1','2'])
      9 graph = graphviz.Source(dot_data)
     10 graph

1 frames
/usr/local/lib/python3.6/dist-packages/sklearn/tree/_export.py in export(self, decision_tree)
    393         # n_features_ in the decision_tree
    394         if self.feature_names is not None:
--> 395             if len(self.feature_names) != decision_tree.n_features_:
    396                 raise ValueError("Length of feature_names, %d "
    397                                  "does not match number of features, %d"

AttributeError: 'GridSearchCV' object has no attribute 'n_features_'

code for decision-tree based on GridSearchCV

dtc=DecisionTreeClassifier()

#use gridsearch to test all values for n_neighbors
dtc_gscv = gsc(dtc, parameter_grid, cv=5,scoring='accuracy',n_jobs=-1)

#fit model to data
dtc_gscv.fit(x_train,y_train)

One solution is taking the best parameters from gridsearchCV and then form a decision tree with those parameters and plot the tree.

However is there any way to print the decision-tree based on GridSearchCV.


texts:
What is(are) the difference(s) between parallel and distributed computing? When it comes to scalability and efficiency, it is very common to see solutions dealing with computations in clusters of machines, and sometimes it is referred to as a parallel processing, or as distributed processing.

In a certain way, the computation seems to be always parallel, since there are things running concurrently. But is the distributed computation simply related to the use of more than one machine, or are there any further specificities that distinguishes these two kinds of processing? Wouldn't it be redundant to say, for example, that a computation is parallel AND distributed?


texts:
Given data:

R   G   B   Color
0   0   0   Black
255 255 255 White
255 0   0   Red
0   255 0   Lime
0   0   255 Blue
255 255 0   Yellow
0   255 255 Cyan_Aqua
...

Can we predict the color given an RGB input? For example,

224, 255, 255=light_cyan

The goal is to generate logical names and not random names. For instance, if the data contains "green", a closest match with lighter hue, would be named "light green".

If yes, some pointers would be very helpful.


texts: How to deal with an attribute including a lot of character or factor values when using caret package in R

texts:
I am studying caret package in R. I know in this package, all the data are assumed to be numeric. But I always have some data that has a lot of strings or factors. For example, a product category could include hundreds of thundred of strings indicating all the different products. A worker ID list may includes a lot of numbers. They look like numeric, but you cannot do any calculation because it does not make any sense. They are actually supposed to be characters or strings. So, we cannot use model.matrix or dummyVars to transfer them to numeric because 1). It should not be transferred like this; or 2). The matrix will be so huge after transferring that we cannot deal with. So my question is:
How can we deal with an attribute with a lot of characters or a lot of so-called numbers when using caret package in R?


texts: User identification using machine learning

texts:
I am quite new to machine learning, and the task I was given was to find attributes that will allow me to identify users.

I have a set of data for training that contains roughly 48,000 instances, where every instance is labeled with user id.
The data are TCP sessions with data on destination, source, ttl, start time, endtime, destination IP addresses, and more.

My task is to find attributes so that in given a difference TCP flows in the same format form the same group of users, I will be able to labeled them with the appropriate user Id.

There are 10 users in the data set.

My question is what the best way to do it, I tried using weka to run number of machine learning algorithm, but I couldn't configure them right, for example I tried to use K-means but the results didn't fit with the labeled user id.


texts: How to combine data having similar distribution?

texts: I have a collection of time series data with data points of around 2 years of daily data. I am thinking of a way to increase the number of data points in it so that the neural network gets a better understanding of the fluctuations in the data. I am suggesting a hypothesis where I try to cluster similar time-series data following similar distribution, in order to increase the number of data points fed into the neural network. Is this a correct way to approach the problem? If so, on what basis do I combine these similar data together so that I can feed it into the neural network/model?


texts:
Hello folks,
            I am trying to plot a grouped bar plot of two variables with varying lengths, which means the x and y length of both the variables are different. The format of the data is given below:
This is for NRI.

This is for RI.

I want these two datasets to be grouped together. When I try to plot it both the datasets are overlapping each other. If anyone can help me in this regard it will be much appreciated.

Here is the code I used:

import numpy as np
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import matplotlib.path as mpath
from PIL import *
import os
import sys
import csv
from matplotlib import rc, rcParams
import pandas as pd
from matplotlib.ticker import (MultipleLocator, FormatStrFormatter,
                               AutoMinorLocator)
#df = pd.read_csv('E:/v.csv')
df = pd.read_excel('IC12_freq.xlsx',sheet_name='NRI')
df_1 = pd.read_excel('IC12_freq.xlsx',sheet_name='RI')

x = df.IC12.values
x1 = df_1.IC12.values
y = df.FRQ.values
y1 = df_1.FRQ.values
fig, ax = plt.subplots(figsize=(10,10))
ax.bar( x+1.3,y,color = 'w', width = 1.3,hatch='***',edgecolor='k',label='NRI',align='center')
#ax.twinx()
ax.bar( x1,y1,color = 'w', width = 1.3,hatch='/////',edgecolor='k',label='RI',align='center')
#ax.plot(x, y,'ro',color = 'k')
#ax.plot(x1, y1,'ro',color = 'r')
ax.xaxis.set_major_locator(MultipleLocator(10))
ax.xaxis.set_major_formatter(FormatStrFormatter('%d'))
ax.xaxis.set_minor_locator(MultipleLocator(5))
ax.xaxis.set_minor_formatter(FormatStrFormatter('%d'))
# For the minor ticks, use no labels; default NullFormatter.
ax.yaxis.set_major_locator(MultipleLocator(10))
ax.yaxis.set_major_formatter(FormatStrFormatter('%d'))
ax.yaxis.set_minor_locator(MultipleLocator(5))
ax.yaxis.set_minor_formatter(FormatStrFormatter('%d'))

plt.rcParams["font.weight"] = "bold"
axis_font = {'fontname':'Arial', 'size':'14'}
tick_font = {'fontname':'Arial', 'size':'5'}

ax.tick_params(which='both', width=2)
ax.tick_params(which='major', length=7)
ax.tick_params(which='minor', length=7)
plt.xlabel('IC12(kt)', fontweight='bold',**axis_font)
#plt.xticklabel(**tick_font)
plt.ylabel('Frequency(%)', fontweight='bold',**axis_font)
ax.set_facecolor("#f1f1f1")
ax.spines['top'].set_linewidth(1.5)
ax.spines['right'].set_linewidth(1.5)
ax.spines['bottom'].set_linewidth(1.5)
ax.spines['left'].set_linewidth(1.5)
leg = ax.legend()

#plt.grid(True)
#plt.style.use('ggplot')
##plt.xlabel("x axis", **axis_font)
#plt.ylabel("y axis", **axis_font)
#plt.bar(y,x)
#plt.savefig('IC12_frq.tif', bbox_inches='tight', dpi=300)
plt.show


texts: Tensor Decomposition for Higher-Order Context-Aware Recommender Systems

texts:
Let me motivate my problem with an example. Let's assume our observations concern ratings a user give to different items, while navigating through item catalog. The user begins rating item1 (may be relevance), and proceeds to item2 to give (or not) a rating to that item. The user may return back to item1 to give yet another score (recursive nature). I am interested in learning a recommender (say, collaborative) which predicts a score based on the context (traversal path) (s)he is in (or learn some kind of a general transfer function which predicts based on past context).

A use case would be to find the true user-item rating score. Another would be to recommend an item to a user with the highest rated score based on the contexts (traversal history) of that user.

The observations can be encoded as:
$$
User \times Context_1 \leftarrow Score \\
User \times Context_1 \times Context_2 \leftarrow Score \\
\vdots \\
User \times Context_1 \times \ldots \times Context_N \leftarrow Score
$$
where any of the two contexts can be the same, eg. $(u_1, it_1, it_2, it_1) = 5$

I have been looking at Tensors to incorporate contextual information for collaborative filtering. On one hand, the resulting tensor has very high number of dimensions (longest possible traversal path of the observed data) and extremely sparse. On the other hand, since the length of traversal history for each user is different, the resulting tensor for unobserved dimensions is incomplete.

$$
User \times Context_1 \times Score_1 \times ... \times Context_N \times Score_N \leftarrow \{0,1\}
$$

Is the tensor decomposition the right technique to tackle this problem? Are deep models such as RNNs better fit for this kind of nested recursive contextual problems.
If TD is the way to go, what is the best approach to deal with the dimensions of the problem? I do know there has been some new techniques such as Tensor-Train or Hierarchical Tucker decomposition for dealing with the `curse of dimensionality'.

Any insights into this problem or pointers is highly appreciated.


texts: What is Coarse-to-Fine in the context of neural networks?

texts: I read in many paper that mentions coarse-to-fine as a technique in deep learning, but I could never figure what exactly they mean. Is it related to multiscale inference, where they use coarse and fine input images?


texts: How to deal with different audio formats for audio classification?

texts:
I am working on an audio classification problem statement to classify between two audio classes. I have collected samples from jotform, they are providing audio widget to collect .wav audio but it turned out that widget is storing data in .mp3 format :
In my problem statement, Classification classes are from different formats :
class A : all the 100 samples are in .mp3 format ( jot form collection )
class B : all the samples are in .wav format

I am adding both types of classes' sample here :
Class A sample audio  : it's in .wav format

Details :

General
Complete name                            : count_class_1.wav
Format                                   : Wave
File size                                : 1.41 MiB
Duration                                 : 15 s 445 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 768 kb/s

Audio
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 15 s 445 ms
Bit rate mode                            : Constant
Bit rate                                 : 768 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 1.41 MiB (100%)

Class B sample audio Jotform says it's .wav format but only extension is .wav, file is .mp3 format.
Details :
General
Complete name                            : count.wav
Format                                   : MPEG Audio
File size                                : 183 KiB
Duration                                 : 9 s 360 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 160 kb/s
Writing library                          : LAME3.99.5
FileExtension_Invalid                    : m1a mpa mpa1 mp1 m2a mpa2 mp2 mp3

Audio
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 3
Format settings                          : Joint stereo / MS Stereo
Duration                                 : 9 s 360 ms
Bit rate mode                            : Constant
Bit rate                                 : 160 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Frame rate                               : 41.667 FPS (1152 SPF)
Compression mode                         : Lossy
Stream size                              : 183 KiB (100%)
Writing library                          : LAME3.99.5

What i am doing before feeding it to neural network :

Downsampled to 16kHz,  the level of the signal was normalized
Segmented in audio segments, by removing the silences in the signal
High filtered (pre-emphasis filter). Audio segments were then
divided    in non-overlapping Hamming-windowed frames of 25ms.

Now after this extracting various features from each frames including MFCCs, Zero-crossing rate (ZCR), Formants (the first 4) etc and at last feeding all these features to simple dense layer neural network or CNN (spectrogram format).
But the problem is both classes' audio files are in a different format class A audio samples are in .wav and class B audio samples in .mp3 and there are high chances that network can be biased towards format or audio encoding.
Solutions I have thought :

Downgrade all files to 16kHz frequency ( But format issue is still there)
or convert all files into one universal format, for example I am converting all .mp3 files to .wav files then all files will be having same format, I could convert one into another, but I am afraid I will lose quality on the converted files.

My doubt is if I downsampled both classes audio samples (.wav and mp3 both) to 16kHz will my neural network still be format biased?
What would be a good strategy for me for Audio classification when audio files are in different formats?


texts: Are there any existing model weights for buildings segmentation from aerial images?

texts:
I'd like to test some deep learning techniques to extract buildings footprint from aerial imagery. I've found many references related to this problem (here, or here), but only providing the model architecture and the process of learning.

I don't have easy access to hardware material in order to train a model. So I was looking for existing model weights (hdf5 or onnx files from state-of-the-art works) or pre-packaged script I could directly use as-is, just to make tests.

Do you know any repository or library with such model weights provided ?

I don't need to classify objects on the image, but only identify buildings (binary classification).


texts: Generative adversarial network error in training process

texts:
I'm trying to make GAN which will generate art from random noise. I rely on this article https://towardsdatascience.com/generating-modern-arts-using-generative-adversarial-network-gan-on-spell-39f67f83c7b4
First off i've created training_data.npy file. For this purpose i wrote simple script.
import os
import numpy as np
from PIL import Image
import sys

args = sys.argv
del args[0]

if not args:
    print('Add at least 1 dir')
    sys.exit(0)

for i in args:
    if i not in os.listdir(os.getcwd()):
        print(f"Can't find {i} here")
        sys.exit()

IMAGE_SIZE = 128
IMAGE_CHANNELS = 3

training_data = []

print('resizing...')
for dir in args:
    for filename in os.listdir(os.path.abspath(dir)):
        path = os.path.join(os.getcwd(), dir, filename)
        image = Image.open(path).resize((IMAGE_SIZE, IMAGE_SIZE), Image.ANTIALIAS)
        training_data.append(np.asarray(image))

training_data = np.reshape(
    training_data, (-1, IMAGE_SIZE, IMAGE_SIZE, IMAGE_CHANNELS))
training_data = training_data / 127.5 - 1

print('saving file...')

def get_name():
    i = 1
    while True:
        if f'training_data_{i}.npy' in os.getcwd():
            i += 1
            continue
        return f'training_data_{i}.npy'

name = get_name()
print(f'saving to file {name}')
np.save(name, training_data)

There are folders(which contain images) in my current directory
After that i wrote code bellow for training model(this code locate in google colab, training_data.npy i also uploaded to google drive and use it in google colab)
https://colab.research.google.com/drive/1pLDdyRwQQe1SqYNUI8pwzgnOeUGrxnR6?usp=sharing
If u click on link above, u'll see my error which i got. Please, help me to resolve it.
Thank u.
P.S
5 days ago i read this article carefully , and understood it. But now i've decided to just copy and paste the code from article and don't reread it again. Forgive me if i make some errors in the code.


texts: Which Regression Machine Learning algorithm to use in Azure for statistical prediction/forecasting

texts:
I've a golf website that pulls a load of statistical data and makes forecasts for upcoming events.

I'm trying to improve the algorithm and have been looking into Azure's ML Studio. I've lots of data about previous events and stats the competing golfers had and the time of the event so I'm confident I need to be using some sort of regression based algorithm but I'm not sure which one, there's a few that look appropriate?

Any advice would be greatly appreciated!
 Thanks


texts: What is the neural network architecture behind Facebook's Starspace model?

texts:
Recently, Facebook released a paper concerning a general purpose neural embedding model called StarSpace.

In their paper, they explain the loss function and the training procedure of the model, but they don't emphasize much on the architecture of the model.

Does somebody know what the NN behind looks like?


texts: How to model product popularity by their prices and sales volume?

texts:
I have a bunch of sales data of similar products and I want to know which product might be the best popular one.

However, for the same category of items, there are cheap ones and expensive ones, and the sales volume of the cheap ones are always higher than the expensive ones.

For instance, we have some ceiling lamps which cost 45\$, and some are 200\$. If the sales volumes for the 45\$ ones and 200$ are 200 and 180 respectively, even the cheap one is sold more than the expensive one. We know that the expensive one is actually more popular then the cheap one.

How can I model the popularity by this kind of information?


texts: Is it mandatory to have a DateTime index for a pandas dataframe to run ARIMA models on it?

texts: I'm trying to learn about Arima models. In every tutorial, the Dataframe considered has a DateTime index. When I looked upon the mathematical formula for Arima model, it only depends on the past observations but not on date time. So is it mandatory to have a datetime column for running arima models on a pandas dataframe?


texts: How to predict correct text from incorrect text with machine learning?

texts:
I have a dataframe like this:

import pandas as pd

df = pd.DataFrame({'incorrect': ['jak', 'mya', 'kfka'],
                   'correct': ['jack', 'maya', 'kafka']})

incorrect  correct
jak        jack
mya        maya
kfka       kafka

I want to predict correct results from incorrect values with machine learning in python. Which model can I use for this case? Thanks.


texts: Loss function for multi-class classifiction where output variable is a level i.e the various classes are dependent on each other

texts:
Let's say we are classifying Images of cat , fish and human. Classifying a cat as human is as wrong as classifying it as fish, so here the normal loss functions/ metrics like Confusion matrix is fine.

But in the case of classifying a baby, child, teen, man, old-man. Classifying a child as old man is more wrong than classifying as a teen. So what would be a better metric for a ranking kind of classification?


texts: How to predict consumer purchase in next 6 months?

texts:
I'm working on a model to predict a customer as being 'in-market' for a product in the next 6 months. The dataset has a wealth of information like lifestyle and demographic variables and previous transaction history. What is the best model to combine the demographic and lifestyle data with transaction history to predict a purchase in the next 6 months? Would a combination of a model that predicts likelihood to purchase with something like a survival model to predict when that purchase might occur work?

Are there any real world examples of something similar to this? Basically predicting time to purchase.


texts:
I want to train bert for a custom entity, and wanted to confirm the correct input format.
The common dataset run is the coNLL, which is formatted like this:
-DOCSTART- -X- -X- O

SOCCER NN B-NP O
- : O O
JAPAN NNP B-NP B-LOC
GET VB B-VP O
LUCKY NNP B-NP O
WIN NNP I-NP O
, , O O
CHINA NNP B-NP B-PER
IN IN B-PP O
SURPRISE DT B-NP O
DEFEAT NN I-NP O
. . O O

I want to format my data in this way, with only a single entity:
& O
gt O
; O
* O
Police O
last O
week O
evacuated O
80 O
villagers O
from O
Waltengoo B-LOC
Nar I-LOC
where O
dozens O
were O
killed O
after O
a O
series O
of O

My questions are:

Does bert use the POS tags at all?

Would it be as simple as formatting my dataset with only the BIO tags, and running run_ner.py from this directory, but changing the labels on line 157 to my custom labels?

Any advice or help on this would be very much appreciated! Especially if you've trained bert on custom entities and used a different workflow.


texts: Knowledge of statistics To be a data scientist

texts:
I am an undergraduate student pursuing my BSc in Maths. I have taken statistics classes in first year. I want to know how much statistics do I need to know to get a decent place in the data science world?

Also, please do provide me with suggestions where to begin if I start my preparation now.


texts: How do I split number string with digit pattern?

texts:
I am trying to split number string to two to digit numbers

How do I get two different Numbers out that string

Example:

I want separate two numbers

x <- c("-26755.22-50150.60")

To this

-26755.22
-50150.60

I have tried stringr::str_split but I din't manage to keep the digits.


texts: Predict next series of numbers based on previous data

texts:
I have the following list of numbers (1 – 5  categories, first row) that I am trying to predict the next sequence for each column (total column sum = 6). There are over 1,500 lines of data in total. I have tried to make a column of the accumulated sum of each line and then use Linear Regression and Time Series Forecasting to predict the next accumulated sum and then subtract the previous number to predict the number (0,1,2,3,4) but I am not getting the results that I am after.
How best can I forecast the next series of numbers using a non-Linear approach ?


texts: T-SNE good clustering but SVM classification poor

texts:
I am trying to classify in 4 different classes, paragraph embedding vector computed with doc2vec using an non-linear svm over them.
When I visualize the embeddings using tensorboard t-sne I can see that they are clustered quite well as in the image.

However, when I train the svm (with rbf kernel and grid search) I obtain an f1-score of 60% that given the figure seems quite low.
Is it common to obtain good cluster with t-sne and bad results with svm?


texts: Taking huge time to execute piepeline text classification model using sklearn?

texts:
I have created a pipeline  model for text classification using python ,Firstly i have tried on 30k records dataset it is working fine got the good results , but when it comes to huge data set like 50k or 1 laksh record data set not at all giving at any results in console even though it is taken more than 24 hours time ?
please suggest  will pipeline model  taking to execute  on huge data for text classification ?
i am not getting that what was the problem with pipeline text classification .

tfidfVector = TfidfVectorizer(tokenizer = textProfilerObj.textTokenizer)

    # Create the  pipeline  model to clean, tokenize, vectorize, and classify using tfidVector

    pipe = Pipeline([("cleaner", TextCleaner()),('vectorizer', tfidfVector),('classifier', modelObj)])

    #fit the data using pipeline

    baypit=pipe.fit(self.trainX,self.trainY)


texts: How to deal with a constant value as an output from neural network?

texts:
I am using feedforward neural network for regression and what I get as a result of prediction is a constant value visible on the graph below:

Data I use are typical standardised tabular numbers. The architecture is as follows:

model.add(Dense(units=512, activation='relu', input_shape=x_train.shape[1],)))
model.add(Dropout(0.2))
model.add(Dense(units=512, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=256, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=1))

adam = optimizers.Adam(lr=0.1)

model.compile(loss='mean_squared_error', optimizer=adam)

reduce_lr = ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.9,
    patience=10,
    min_lr=0.0001,
    verbose=1)

tensorboard = TensorBoard(log_dir="logs\{}".format(NAME))

history = model.fit(
    x_train,
    y_train,
    epochs=500,
    verbose=10,
    batch_size=128,
    callbacks=[reduce_lr, tensorboard],
    validation_split=0.1)

It seems for me that all weights are zeroed and only constant bias is present here, since for different data samples from a test set I get the same value.

I understand that the algorithm has found smallest MSE for such a constant value, but is there a way of avoiding such situation, since straight line is not really good solution for my project?

[EDIT]
Adding learning curves for trainging and validation sets.


texts:
I'm from the vision world and only worked with pixels from 0-255, ignoring any side effects. My current problem is different, in the way that I cannot rely on the input data.

What my problem is:

I have a number of inputs. Each input is categorical (for now) and optional. For example I have a number of user features, {male, female, [..not given]}, {single, relationship, … , [..not given]}, ..

What I want:

X optional Input-Features mapped to Y Output-Features, with uniform output vales across the features.

I tried kernel pca, different kinds of matric factorization for guessing missing inputs and simple autoencoder networks. By sight, the last two yield "ok" results. Any advice here?


texts:
I am trying to predict sales quantity based on attributes of the item.Sales are aggregated by week wise and prediction is also done by week wise.I am having large number records with zero sales quantity than compared to positive sales(for 20 positive sales 250 zero sales records are there).I want to increase my training data by adding small values to sales quantity and combining with total records.
At present I am using RANDOM FOREST and NEURAL NETWORK .I am not getting any good results
Please correct me if anything is not considerable


texts: How to properly predict date using Orange 3

texts:
First of all - I'm new to all of this.

I'm trying to create a model to predict the release of ios 12 based on previous years. I've got an excel that has a format like this:
ios version | release name | date

Each version consists of about 5 betas + general release.
I've set up the import like this:

Secondly I've set up flow like this:

First step selects all the rows that have [date] param - those go to either kNN or Linear Regression, everything else (1 row with ios 12 release date) goes to prediction and then to a table.

Cool, now depending on the modeling I get different results (as expected). It's either 2015-09-17 (kNN) or 2018-09-10 (LR). I've used the Test & Score, which gives me:

If I'm reading this properly - Linear Regression is more accurate (R2 column), meaning that iOS 12 will release on 10th of Sept 2018  (DUH! not 2015).

But if I look at release dates up to now, this might be wrong, because no other release happened so soon:

So I also did an exercise where I've adjusted the dates, so that they are all in 2018 (so that the year is less important, and I'd care more about relation within each year, given the features) - here are predicted results:

and Test & Score for the offsets:

Now... I would welcome all comments.
Am I using the wrong modeling? Something other than kNN & LR? Should I be using offsets? Am I using the tool completely incorrectly?


texts: Has the Random Forest algorithm ever been used in Reinforcement Learning applications?

texts:
I've seen a research paper describing a "Reinforcement Learning Tree", which the authors say that it has a better convergence than random forests.

However, I couldn't find anything related to utilizing random forests for a table in Reinforcement Learning. Have tests ever been conducted on using random forests in reinforcement learning?


texts: How to include validation set in the pipeline to tune parameters for an SVM?

texts:
I have a dataset already divided into train, test and validation set. How can I insert the validation in my pipeline?
Code:
class SVMSentiment(Base):
"""Predict sentiment scores using a linear Support Vector Machine (SVM).
Uses a sklearn pipeline.
"""
def __init__(self, model_file: str=None) -> None:
    super().__init__()
    # pip install sklearn
    from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer
    from sklearn.linear_model import SGDClassifier
    from sklearn.svm import LinearSVC
    from sklearn.pipeline import Pipeline
    self.pipeline = Pipeline(
        [
            #('vect', CountVectorizer()),
            #('tfidf', TfidfTransformer()),
            ('tfidf', TfidfVectorizer()),
            #('clf', LinearSVC(
                #loss='hinge',

            ('clf', SGDClassifier(
                loss='hinge',
                penalty='l2',
                alpha=1e-4,
                random_state=42,
                max_iter=100,
                learning_rate='optimal',
                tol=None,

            )),
        ]
    )

def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
    "Train model using sklearn pipeline"
    train_df = self.read_data(train_file, lower_case)
    dev_df = self.read_data(dev_file, lower_case)
    learner = self.pipeline.fit(train_df['text'], train_df['truth'])
    # Fit the learner to the test data
    test_df = self.read_data(test_file, lower_case)
    test_df['pred'] = learner.predict(test_df['text'])
    return test_df

I don't understand where I should include it since the data is already splitted.
Edit: more on the predict method, I see dev is not contemplated here:
def run_classifier(files: Tuple[str, str, str],
               method: str,
               method_class: Base,
               model_file: str,
               lower_case: bool) -> None:
"Inherit classes from classifiers.py and apply the predict/accuracy methods"
train, dev, test = files  # Unpack train, dev and test filenames
result = method_class.predict(train, test, lower_case)
method_class.accuracy(result)
# Plot confusion matrix
make_dirs("Plots")
print(result)
fig, ax = plot_confusion_matrix(result['truth'], result['pred'], normalize=True)
ax.set_title("Normalized Confusion Matrix: {}".format(method.title()))
fig.tight_layout()
fig.savefig("Plots/{}.png".format(method))


texts: How to i get word embeddings for out of vocabulary words using a transformer model?

texts:
When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary word/token, that is being split into em,bed,ding,s.
I would like to know if there is any aggregation strategies available that make sense apart from doing a mean of these vectors.
from transformers import AutoTokenizer, AutoModel
# download and load model
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

sentences = ['This framework generates embeddings for each input sentence']

#Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=128, return_tensors='pt')

#Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

print(encoded_input['input_ids'].shape)

Output :
torch.Size([1, 13])
for token in encoded_input['input_ids'][0]:
  print(tokenizer.decode([token]))

Output:
[CLS]
this
framework
generates
em
##bed
##ding
##s
for
each
input
sentence
[SEP]


texts:
I'm working on one use case where I have to explore source code repo files. Different files will be a categorical values for me. But with such large number of files, One Hot Encoding comes out to be very large.
Also, all files are divided among unique modules, such that each files belongs to a specific module.
So if I consider unique files associated with specific modules comes out to be small in number and One Hot Encoding matrix is also not large.

So if I define different modules as a  categorical features, I'm not sure how to map the associated unique files.
How to derive new feature with such combination of two categorical features?
Any other better way to handle such use case?


texts: Multi-Head attention mechanism in transformer and need of feed forward neural network

texts:
After reading the paper, Attention is all you need, I have two questions:

1. What is the need of a multi-head attention mechanism?

The paper says that:

  "Multi-head attention allows the model to jointly attend to
  information from different representation subspaces at different
  positions"

My understanding is that it helps in anaphora resolution. For example:- "The animal didn't cross the street because it was too ..... (tired/wide)". Here "it" can refer to animal or street based on the last word. My doubt is why can't a single attention head learn this link over some time?

2. I also don't understand the need of feed-forward neural network in the encoder module of the transformer.

Thanks for your help.


texts:
I'm trying to find the patterns in the crime records I have in a database.
I thought clustering would be a way to do it.

This is my (cooked up) dataset:

age,nationality,country_of_birth,place_of_birth,no_of_checkedinbaggage,noofcabinbaggage,no_of_co_passengers,watchlist
34,GBR,GBR,London,2,1,0,Drug Trafficker
32,IND,IND,Delhi,2,1,0,Human Trafficker
31,USA,USA,Tampa,2,1,0,Arms Dealer
.....

Basically, I'd want to identify the clusters of watchlists and see if they are having a pattern. Based on the cluster, I'd want to predict future data as well.

Is clustering (K-Means) the correct choice? And also, do all the variables have to be numeric? If so, I'm not sure how I can encode them to numerics. Thoughts?


texts: How to measure Covid impact by analysing credit card transaction of customer

texts:
I Want to know how can I identify that is the customer is in financial distress due to the COVID situation using its credit card transactions.

I have a daily transaction of customers till current date,
Any thoughts or ideas would be really helpful.

For example, my thoughts that if sudden increment in credit card utilization then it can be an indicator that person in financial distress as could be a flag to potential lenders or creditors that  having trouble managing your finances.


texts: What kind of algorithm should I use to build ML model that can predict just next reoccurence of an event in the future (at irregular time interval)?

texts:
I'm quite new to machine learning and statistics.

I've a dataset from some ecommerce sale's history. It's almost 2k instances, and features include personId (string), productCategory (string/discrete), amountPaid (float/continuous), purchaseTime (string/Time(DD/MM/YYYY)). Person can purchase product at any time (irregular time interval so I can't use time series analysis, I guess).

I want to know when will the same person (attr with person Id) make just next purchase in a category (attr with productCategory). What ML model should I use for this problem? Should I use Time Series forcasting or Survival Analysis or maybe some other function? How should I approach this problem?


texts: pytorch code for VAE for MINST forces mu and logvar to zero

texts:
I am new to pytorch and trying to implement a VAE for MNIST data. When I try to train my model, it appears that the model forces mu and logvar to zero (or something very close to zero) independent of the input. In a way it appears that it is failing to take into account the MSE part of the loss function, but I don't understand why.
Here's the complete code I am using:
import torch
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
from torch import nn
import torch.nn.functional as F

batch_size = 32

loc_data = 'MNIST'
transformations = transforms.ToTensor()
mnist_train =  datasets.MNIST(loc_data, train=True, download=True, transform = transformations)
mnist_test = datasets.MNIST(loc_data, train=False, download=True, transform = transformations)

train_loader = DataLoader(mnist_train,
                          batch_size=batch_size,
                          drop_last = True,
                          shuffle=True)
test_loader = DataLoader(mnist_test,
                         batch_size=batch_size,
                         drop_last = True,
                         shuffle=True)

class Encoder(nn.Module):
  def __init__(self, latent_dim=10):
    super(Encoder, self).__init__()
    self.latent_dim = latent_dim
    self._encoder = nn.Sequential(nn.Linear(in_features = 28*28, out_features = 512),
                                  nn.ReLU(),
                                  nn.Linear(in_features = 512, out_features = 2*latent_dim)
                                 )

  def forward(self, x):
    x = torch.reshape(self._encoder.forward(x), (-1, 2, self.latent_dim))
    mu, logvar = x[:,0,:], x[:,1,:]
    return mu, logvar

class Decoder(nn.Module):
  def __init__(self, latent_dim=10):
    super(Decoder, self).__init__()
    self.latent_dim = latent_dim
    self._decoder = nn.Sequential(nn.Linear(in_features = latent_dim, out_features = 512),
                                  nn.ReLU(),
                                  nn.Linear(in_features = 512, out_features = 28*28),
                                  nn.Sigmoid())

  def forward(self,x):
    return self._decoder.forward(x)

def sample(mu, logvar):
  z = torch.randn_like(mu)
  return mu + torch.mul(torch.exp(0.5*logvar), z)

def vae_loss(x, x_hat, mu, logvar):
  mse = (x - x_hat).pow(2).sum()/(x.shape[0]*1.0)
  KL_loss = 0.5*torch.sum(-1 + torch.pow(mu,2) - logvar + torch.exp(logvar))
  return torch.add(mse, KL_loss)

def train(encoder, decoder, train_loader, optimizer, num_epochs = 10):
  encoder.train()
  decoder.train()
  for ii in range(num_epochs):
    print("Epoch {}".format(ii))
    for jj, (x, y) in enumerate(train_loader):
        x = torch.reshape(x, (-1,28*28))
        x.to(device)
        _mu, _logvar = encoder.forward(x)
        _z = sample(_mu, _logvar)
        x_hat = decoder.forward(_z) #.reshape((-1,28,28))
        optimizer.zero_grad()
        loss = vae_loss(x, x_hat, _mu, _logvar)
        loss.backward()
        optimizer.step()
        if jj % 100 == 0:
            print(loss)
  return loss

latent_dim = 20
encoder = Encoder(latent_dim)
decoder = Decoder(latent_dim)
params = list(encoder.parameters())+list(decoder.parameters())
optimizer = optim.Adam(params, lr=1e-2)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

train(encoder, decoder, train_loader, optimizer, num_epochs = 1)

when I try to probe the mu or logvar for some test data, it seems that the result is almost identically zero.


texts: Find effective feature on machine learning classification task with scikit-learn

texts:
I'm tackling a binary classification task using SVM implemented in python scikit-learn. Datasize is around 10,000 and the number of feature is 34.
After finding nice parameter set (using RandomizedSearchCV class), I evaluate the model by the cross validation. The result seems nice.
criteria_list = ["precision", "recall", "f1", "roc_auc"]
score_df = []
score_df2 = []
clf = svm.SVC(**random_search_clf.best_estimator_.get_params())
for crit in criteria_list:
    scores = cross_validation.cross_val_score(clf, X, y, cv=3, scoring=crit)
    score_df.append(["{} (±{})".format(np.round(np.mean(scores),3), np.round(np.std(scores),4)), scores])
    score_df2.append(["{} (±{})".format(np.round(np.mean(scores),3), np.round(np.std(scores),4))])

pd.DataFrame(np.transpose(score_df2), columns=criteria_list, index=["SVM"])

My question is whether it is possible to find out which feature is effective to classify the test data. I thought it's relating to sensitivity analysis, but good answer cannot be shown by googling "sensitivity analysis + svm" or "sensitivity analysis + scikit learn".


texts: Cosine similarity between query and document confusion

texts:
I am going through the Manning book for Information retrieval. Currently I am at the part about cosine similarity. One thing is not clear for me.

Let's say that I have the tf idf vectors for the query and a document. I want to compute the cosine similarity between both vectors.

When I compute the magnitude for the document vector, do I sum the squares of all the terms in the vector or just the terms in the query?

Here is an example : we have user query "cat food beef" .
Lets say its vector is (0,1,0,1,1).( assume there are only 5 directions  in the vector one for each unique word in the query and the document)
We have a document "Beef is delicious"
 Its vector is (1,1,1,0,0). We want to find the cosine similarity between the query and the document vectors.


texts: The MNIST handwritten digit dataset uses a file format .idx. What are the advantages of this file format over alternatives such as CSV, TSX and ODS?


texts:
I am working with a really imbalanced dataset ($\approx$ 1% of positive cases) for a classification problem. I know that class balancing is an important step in this scenario.
I have two questions:

Considering that I don't want to put the 0/1 label, but just to order the record according to the output score (it is always a calibrated probability of being in the positive class), is it still a good idea to do class balancing or, considering the specific output required, it is useless?
Basically, I do not care about the cut-off point, but I just sort the record in order to identify the one with a higher probability of being positives.

Considering the really small percentage of positive cases, is it better to do over/under sampling? Is there any rule-of-thumb to decide the proportion of resampling?

Thank you in advance!


texts: Dummy coding a column in R with multiple levels

texts:
I have a dependent variable measuring the net revenue. One of the major predictor affecting this is "product" i.e. the product sold to the customer. My randomly sampled dataset contains 1.4 million entries.

Products are assigned a specific categorical value.
I feel that using dummy variables to represent the products would be apt however, there are 4481 levels of products. I do not know how to code so many levels in R.

model.matrix(~ product, data=salesdata)
 returns an error. (Needs 38.4GB of memory)

Can someone guide me a little on how to code these categorical variables?

Dependent: Net revenue (quantitative)
Independent: Product code (quantitative but treated as qualitative since values are nominal)


texts: Standard for aligning data to week based on Mon to Sun for forecasting

texts:
I need to produce a forecast where i have predicted volume to a standard definition of a week.

This definition is simply the week starts on Monday and Ends on Sunday.

Is there an ISO standard that already exists which communicates this?

Some more info:
We plan our business according to the Calendar year. So for example we plan out and allocate capacity from 1/1/xxxx to 12/31/xxxx. However I need to align sales data in the past. to do this requires consolidating it accurately so that past weeks can be used to predict future weeks. I'm hoping i can keep the forecasted week definition and past sales week definition the same for modeling input. weeknum=1, 2, 3 ... 52 etc.

I could pick a monday in the past and call all 7 days in that week "week 1". but wondering if there is already a standard which will do this and the assumptions/caveats around it commonly used? I see excel has a weeknum() function. Not sure if it uses a ISO standard etc.


texts:
Cosine similarity vs The Levenshtein distance

I wanted to know what is the difference between them and in what situations they work best?

As per my understanding:

Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them. The cosine of 0° is 1, and it is less than 1 for any angle in the interval (0,π] radians.

The Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits

My question is

When would one use Cosine similarity over The Levenshtein distance?


texts: xgboost - How do I treat document ID in pairwise ranking

texts:
I am trying to use xgboost in R for pairwise ranking for an implicit dataset.

For simplicity, let's assume that I am dealing with a search problem, where I want to rank documents relative to a given query. Each query has an ID and the documents that were clicked also have an ID. My dataset only consists of the positive feedback and looks like this:

queryID    documentID    features    target
   1            1            ...        1
   1            3            ...        1
   1            13           ...        1
   2            3            ...        1
   2            4            ...        1
   3            990          ...        1

I have read, that for grouping my queries together, I have to use setinfo(group, list). But what do I do with my documentID, do I implement it as just another feature in my feature list?


texts: How to Predict/Forecast street's Traffic based on previous values?

texts:
I have a dataset which has the following 5 columns:

date, hour, day_of_week, street_id, counts

My dataset has information about the number of cars that each street (same city) has in a given hour of a certain date, and I want to predict the traffic count that a certain street has in a given hour of a certain date.

I think I could use certain variables depending on the day and hour that I want to predict, for example, if a want to predict the traffic count of a working Wednesday:

Results of others working days
Results of others Wednesdays
...

I want to use Spark MLlib to perform the prediction because I have experience with Spark and I have large datasets.

How you deal with this kind of problem?

Any ideas?


texts: How to transform time series data to apply supervised learning algorithms to it?

texts:
Apologies in advance for what may be a very basic question.

I have a dataset consisting of marketing calls to different clients, which include the timestamp for the call.
My goal is to train a model to predict if whether a customer will answer a call or ignore it based on the call time, as well as other features like caller id, etc.

The issue is that the outcome of the call is clearly time dependent, that is, the order of the data points per client matters for the prediction of the dependent variable at a given time.

My question is, how can I transform my features, so that I can use standard classification algorithms like Logistic Regression/Random Forest to classify a new data point? Are these algorithms effective for these scenarios? If so, how  should I proceed to take the previous data points into consideration?

I have read that time series data can be converted to a supervised learning problem, by including lead and lag columns of the dependent variable. But since my test data will not have those columns, I am confused to how they might help me.

Thank you very much.


texts:
I'd like to get records with country codes not in a long list , something along the lines of

country_list = ['BR','MX','CO',...]
df = df[df['country_code'] not in country_list]

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

what's an elegant way to do it?


texts: Define difference between feature selection and feature reduction

texts:
What is the difference between feature selection and feature reduction?

When do we use feature selection and what happens when we don't use it? How is this different than feature reduction?


texts:
I am trying to use my data to predict the classes of the input. My data are as the following:
x1 = [0.2, 0.25, 0.15, 0.22] y = 1
x2 = [0.124, 0.224, 0.215, 0.095] y = 3
...
xn = [...] y = 2

The problem is that my data are just lists of numbers that do not have an order. I mean that x1 can be x1 = [0.2, 0.25, 0.15, 0.22] y = 1 or x1 = [0.25, 0.22, 0.2, 0.15] y = 1 or the numbers in the list to be in any other order.
Is there anything that I can do, so I will be able to build a classifier? Thank you!


texts: Is there any clear tutorial for how to use AutoEncoders with text as input

texts: I have a pandas dataframe that describes some fields of the register. I have used one hot encoding to encode the feature vectors that are not numbers. Finally my dataset now has 4000 rows * 4 columns. It contains only numbers. I want to generate the same input using AutoEncoders but I didn't find any useful link that I can use for that. The ones that I used had some dimension problems when I use my data. Does Anyone recommend any helpful tutorial ?


texts: What is the name of this statistical interaction?

texts:
What is the name of the following statistical / informational interaction:

given A, I know exactly what B is.
given B, I know to some extent what A is.

I'm not looking for a probability but rather something like correlation.
Something that tells me that I don't need to include B as a feature when A
is already a feature. A correlation heatmap wouldn't give me this information, but there must be some computation that tells me "Don't include B, it's worthless."

To give some intuition: A could be an item_id and B an item_category.

Or am I wrong? Is it not worthless at all?


texts: Recommended Language/Framework for Building a New Recommendation Engine

texts: Next week I'm going to begin prototyping a recommendation engine for work. I've implemented/completed the Netflix Challenge in Java before (for college) but have no real idea what to use for a production/enterprise level recommendation engine. Taking into consideration everything from a standalone programming language to things like Apache Mahout and Neo4j, does anyone have any advice on how to proceed?


texts: Matching documents from different sets with tfidf and cosine distance

texts:
I have two different set of documents S1, S2, with 30 text documents each.
Using some text representation method, such as tfidf and a distance measure, such as cosine similarity, I want to match similar documents from the two sets S1, S2.
For example D1 from S1 is similar (say 0.36 similar ) to D28 from S2.
My problem is that Tfidf.Vectorizer() creates an array of 30, 5000 for S1 and 30, 4500 for S2, with 30 rows for each document and words of all the documents as columns.
If I calculate cosine_similarity=(S1, S2) vectorized products, I will simply get the similarity between sets as a whole, which is not what I am trying here. I am not interested in finding similarity between documents inside the same set.
Question is:
Is there a way to vectorize each document, for each set, on its own and then calculate pairs distance?
Or is there a way to implement the above method, and then find which documents are similar, based on the tfidf matrix of the set, as described above?


texts: What is the intuition behind using Monte Carlo to solve a differential equation

texts:
Conceptually, I understand how a numerical method like Monte Carlo is used to solve a definite integral. Because integral of a function is the area bounded by the curve, the ratio of random points that land inside the curve to the total number of points is the value of the integral.

Conceptually, can someone explain for a non math person, how we can solve a PDE/ODE using Monte Carlo?


texts: How can you build a model that extracts data out from receipts?

texts:
I'm trying to build a model that is capable of identifying information on receipts and invoices.

I have used google cloud vision api for text extraction from the receipt but the problem is it just returns all the text from a receipt. I am looking to build a model that returns only a certain fields such as total price, brought alcohol etc.. from a receipt.

I could parse the text to extract by hard coding things but it's not optimal I think. Is there any way to build something for this use case. I need just something to go on.

Comment --- Output:

'Berghotel\nGrosse Scheidegg\n3818 Grindelwald\nFamilie R. Müller\nRech. Nr. 4572\nBar\n30.07.2007/13:29:17\nTisch 7/01\nNM\n#ರ\n2xLatte Macchiato à 4.50 CHF\n1xGloki\nà 5.00 CHF\n1xSchweinschnitzel à 22.00 CHF\n1xChässpätzli à 18.50 CHF\n#ರ #ರ #1ರ\n5.00\n22.00\n18.50\nTotal:\nCHF\n54.50\nIncl. 7.6% MwSt\n54.50 CHF:\n3.85\nEntspricht in Euro 36.33 EUR\nEs bediente Sie: Ursula\nMwSt Nr. : 430 234\nTel.: 033 853 67 16\nFax.: 033 853 67 19\nE-mail: grossescheidegg@bluewin.ch\n'


texts: Training multi-label classifier with low quality training set

texts:
So I'm creating a topics classifier where a document may be tagged for several different topics, let's say - A, B while actually the document belongs to A, B and C.
In the training stage I want the classifier to learn that the document belongs to A and B but I'm not sure about class C so I don't want it to learn that the document doesn't\does belong to class C.
Any ideas on how to implement such thing?

I thought about adding weights to the output labels (low weight means that there's no way the document belongs to the aspect, high weight means that the document belongs to the aspect for sure and mid weight means I'm not sure (so the penalty in this case will be lower).


texts:
I'm trying to use isolation forest algorithm for outliers detection.
Data has 2 columns: id and REV. Below code gives me ungrouped result.
Could you pls advise how to get result grouped by first column (id)?

data= pd.read_excel (my_path)
outliers_fraction=0.1
scaler = StandardScaler()
np_scaled = scaler.fit_transform(data)
data = pd.DataFrame(np_scaled)
model =IsolationForest(contamination=outliers_fraction)
model.fit(data)
data['anomaly'] = pd.Series(model.predict(data))
print(data)

I have 2 columns: id and REV. I added a picture of what I expect to see as the final result:

Tried to use function, but have half of data NaN. Any thoughts please?

def groupreg(g):
        outliers_fraction = 0.1
        # train isolation forest
        model = IsolationForest(contamination=outliers_fraction)
        model.fit(g)
        g['anomalyX'] = pd.Series(model.predict(g))
        return g

    df1=data2.groupby('id').apply(groupreg)
    print(df1)


texts: Model Tree M5 - Robustness to Data Quality Issues

texts:
I am currently investigating the M5 tree algorithm by Quinlan(1992) link here: https://sci2s.ugr.es/keel/pdf/algorithm/congreso/1992-Quinlan-AI.pdf
An example of a linear regression model of the algorithm can be seen below:

An implementation of the model similar to Scikitlearn can be found here:
https://github.com/ankonzoid/LearningX/tree/master/advanced_ML/model_tree
The M5 model is a more advanced implementation of the standard decision trees such as the IDE3 or C4.5.
Instead of simple binary splits of the training features the data is split to the Standard Deviation Reduction calculated as follows against linear functions at leaf nodes:

At the end the model produces composite linear regression sections for the different split portions of the model at the leaf nodes.
Would the same robustness principles hold for outliers, missing values and noise as for regression trees?


texts: Pearson correlation method using absolute values and relative values

texts:
I have a dataset with election results and crime rates per city. For each variable I have an absolute value (i.e. Total votes, Total crimes) and a relative value (i.e. Percentage shares of votes).

I want to calculate the correlation coefficient for some variables, but in the process I had a question about what value I need to use, if relative values or absolute values.

First I calculated z score for absolute values and then I calculated the correlation using excel. I also used pandas.DataFrame.corr() and pearsonr from scipy.stats.stats in python, in order to corroborate results.

For example, if I use absolute values I will get a positive correlation between candidate 1 and candidate 2.

x = df['Abs Cand 1'].tolist()
y = df['Abs Cand 2'].tolist()

print (pearsonr(x,y))
(0.95209664861187004, 0.0)

However, if I use relative ones I will get a negative correlation:

x = df['Rel Cand 1'].tolist()
y = df['Rel Cand 2'].tolist()

print (pearsonr(x,y))
(-0.99704737036262991, 0.0)

I was confused when I saw both results, and now I need some orientation to understand those differences.

Thanks in advance!


texts:
Can someone tell me how to integrate PyMC3 with GPU? I’m running a Bayesian regression model and using ADVI method.
Your help is appreciated.


texts: IIoT Sound Classification with Little Data

texts:
[Problem Statement] I have been working on a sound a classification problem with less 200 sound files (wav format) with the following imbalanced class distribution:
Class1 110
Class2 50
Class3 20
Class4 12

Here are Time-Amplitude representation based on librosa.display.waveplot for a random sample from each class:

At present I am either over-fitting or have a low train/test accuracy. I am aware the more training data the better esp. for Class3 and Class4, however, since data labelling for this task is an expensive process, I am exploring ways to avoid that as much as I can, and still obtain a decent model (e.g. a model with about 90% validation accuracy) with data at hand. Sounds are from a machine in a manufacturing line, thus naturally noisy and with varying lengths for each Class.
[Attempts] So far I have tried the followings:

Mel-Spectrogram representation:
I have converted audio files to log-scale Mel-Spectrogram (basically images as shown below) and performed a multi-class image classification, as well as transfer learning. Also I have used Google AutoML Vision to conduct transfer learning. Whatever I do, I cannot do better than 60-65% accuracy.

Mel-frequency cepstral coefficients (MFCCs):
I have tried to extract MFCCs features using librosa as below:

audio, sample_rate = librosa.load(file, res_type='kaiser_fast')
mfccs = librosa.feature.mfcc(y=audio, sr=sample_rate, n_mfcc=40)
mfccsscaled = np.mean(mfccs.T,axis=0)

and feed extracted features into a Fully Connected NN (FCNN) or CNN. YET I have facing not-easily-solvable over-fitting (things I have looked at: model architecture e.g. very simple to complex, regularization via l1 or l2 regularizers, class weight, feature scaling, batch normalizations, weight initializers, dropout, optimizers, change of loss, learning rate more and more), still I get somewhat like:

Most likely due to this over-fitting, I am facing a known only-one-class prediction (meaning NN mostly predicts one class), and strangely, this class is in fact a minority Class3!
[Questions]

Are there any other methods like Transfer Learning for Images but for Audios?
How about WaveNet, and directly train on Audio files? I cannot seem to find an official implementation, or at least something reliable. Anyone experienced using WaveNet?
Any recommendations how to make this model somehow better? I am leaning towards the fact it is within the nature of the audios, that is hard to distinguish, still I don't get it how model can be fooled on the training set up to 100% accuracy then (well accuracy might be a good indicator as well).

I have looked/am still looking into many posts, examples, suggestions to tackle this problem (list is too long, I wouldn't list them here), but rather curious if there are anything that I fundamentally miss here. Maybe simply I gotta stop and collect more data at any costs (although I doubt that would solve the problem completely), what if noise is my enemy!
[Last but not least, Data Collection] It may a bit biased towards certain samples/classes. Image below shows a schematic view how it is done at the moment:

Blue shows a box where certain items (shown in black) are inserted. Microphone that is placed at the top side of the box records each of these black-item placement for inspection. However, it is noted that distance from Microphone varies to the items by 5-10 cm! For example, black items on the left may all belong to Class2, but still produce a slightly different sound wave due to their varying position to the Mic. This may be negligible, but cannot be just yet ruled out!


texts: Optimizing a model for three different metrics that have different ranges

texts:
I have a multiple object tracker that I apply on a specific object in an image series. The tracker has several parameters that can be adjusted which affects the performance of the tracking. I am testing the tracker on ground truth data and evaluate tracking performance by three different metrics:

How close are the number of objects that the tracker outputs to the actual (ground truth) number of objects in the series. I want this to be as close the the actual value as possible. The value is an integer fixed to each series.
How many false positive does the model predict. If the model fails to associate a detection to a group, it will consider it a false positive. However, there are no actual false positives in the test data, so this number should be as close to 0 as possible.
The MOTA score. This score counts the number of mismatches the tracker made. The score ranges from -Inf to 1. 1 is a perfect score.

I have the idea that I would like to test out a large number of combinations of parameters in the tracking model (automatically, in Python) and then find the combination of parameters that give the overall best score of the three metrics above.
Let's say I want to give the three metrics the same weight. How would you do this?
I am thinking that I need to standardise the values in some way, for example to be between 0 and 1.
My idea so far is to calculate metric 1 as:
|1−(Nt/Na)|
Where Nt is the number of objects returned by the tracker and Na is the actual number of objects
Metric 2 could be:
FPt/P
Where FPt is the number of false positives predicted by the tracker and P is the actual number of positives (sum of all detections).
Metric 3:
I am not sure how to standardise this to a value between 0 and 1, where 0 is perfect.
Assuming that I have the last metric standardised, I would then sum these three scores and find the combination of metrics that give the sum closest to 0. Is this a viable approach to the problem?


texts: What is the dimensionality of the bias term in neural networks?

texts:
I am trying to build a neural network (3 layers, 1 hidden) in Python on the classic Titanic dataset.
I want to include a bias term following Siraj's examples, and the 3Blue1Brown tutorials to update the bias by backpropagation, but I know my dimensionality is wrong. (I feel I am updating the biases incorrectly which is causing the incorrect dimensionality)
The while loop in the code below works for a training dataset, where the node products and biases have the same dimension, but once I pass in a test example into the predict function, the dimensions do not match up and get an error. I have commented my code with the dimensions of the calculations of dot products between nodes and inputs.
Can someone help me understand what the dimensionality of the bias term should be, both in this particular case and in general, and how it should be added (row-wise, column-wise)?
Code:
def sigmoid(x, deriv=False):
    """
    Activation function
    """
    if(deriv==True):
        return (x*(1-x))
    return 1/(1+np.exp(-x))

# learning rate, hidden layer dimension, error threshold, dropout rate
alpha, hidden_size, threshold, drop_rate = (0.035,32,0.1,0.5)

# x_train and y_train are the training dataset and corresponding classes

# syn0 and syn1 are the synapses, weight matrices between layers (3 layers, 2 synpases)
syn0 = 2*np.random.random((x_train.shape[1],hidden_size)) - 1 # NxH
syn1 = 2*np.random.random((hidden_size,1)) - 1 # Hx1
b1 = np.random.random((x_train.shape[0],hidden_size)) # MxH
b2 = np.random.random((x_train.shape[0],1)) # Mx1

layer_2_error = 100*np.abs(np.random.random((y_train.shape[0],1))) - 1 # Mx1
avg_err = []
count = 0

while np.mean(np.abs(layer_2_error)) > threshold:

    # Forward
    layer_0 = x_train # training dataset
    A = np.dot(layer_0,syn0) + b1 # MxN X NxH + MxH ~ MxH
    layer_1 = sigmoid(A)
    # drop out to reduce overfitting
    layer_1 *= np.random.binomial([np.ones((len(x_train),hidden_size))],1-drop_rate)[0] * (1/(1-drop_rate))

    B = np.dot(layer_1,syn1) + b2 # MxH X Hx1 + Mx1 ~ Mx1
    layer_2 = sigmoid(B)

    # Backprop
    layer_2_error = layer_2 - y_train # Mx1
    layer_2_delta = layer_2_error * sigmoid(layer_2,deriv=True) # Mx1 * Mx1 ~ Mx1

    layer_1_error = np.dot(layer_2_delta,syn1.T) # Mx1 X 1xH ~ MxH
    layer_1_delta = layer_1_error * sigmoid(layer_1,deriv=True) # MxH * MxH ~ MxH

    # update weights
    syn1 -= alpha*np.dot(layer_1.T,layer_2_delta) # HxM X Mx1 ~ Hx1
    syn0 -= alpha*np.dot(layer_0.T,layer_1_delta) # NxM X MxH ~ NxH

    # update biases
    b2 -= alpha*layer_2_delta # Mx1
    b1 -= alpha*layer_1_delta # MxH

    avg_err.append(np.mean(np.abs(layer_2_error)))
    if count % 500 == 0:
        print("Error after",count,"iterations:",np.mean(np.abs(layer_2_error)))

    count += 1

def predict(x, w0, w1, b1, b2):
    """
    Function to predict an output given a data x, weight matrices w1 & w1 and biases b1 & b2
    """
    A = np.dot(x,w0) + b1 # mXN X NxH (+ MxH) ~ mxH
    layer_1 = sigmoid(A)
    B = np.dot(layer_1,w1) + b2 # mxH X Hx1 (+ Mx1) ~ mx1 (preds)
    layer_2 = B
    return (sigmoid(layer_2) > 0.5).astype(int)


texts: Interpretation of an SVD for recommender systems

texts:
The idea is to motivate the SVD for use in a recommender system.

Consider a matrix $A\in \mathbb{R}^{f\times u}$ where $A_{ij}$ caputures how user $j$ rates film $i$ (on a scale from 1-10, some entries may be missing).
Considering matrices $K=AA^T$, $L = A^TA$ what do $K_{ij}$ and $L_{ij}$ tell us?
What interpretations can you give for matrices $U$ and $V$ in the SVD $A=U\Sigma V^T$?

Riiight ... so
K = $AA^T$ = $U\Sigma^2U^T$, L = $A^TA = V\Sigma^2V^T$
With $K_{ij} = \langle A[i,:], A[j,:] \rangle$ being the dot product of all ratings for film $i$ with all ratings for film $j$.
Similarly, $L_{ij} = \langle A[:, i], A[:,j] \rangle$ being the dot product of all ratings from user $i$ with all ratings from user $j$.
That tells us ... what exactly? I expect it to amount to some kind of similarity measure, but beyond that, I have no idea.
As for $U$ and $V$ ...  I know the first $r$ of them to be Eigenvectors of $K$ and $L$ for some $r$. I also know
$u_1,..,u_r$ is an orthonormal basis for the column space
$v_1,..v_r$ is an orthonormal basis for the row space
but that doesn't really tell me anything.
Yet have no idea if the notion of "row space" and "column space" even makes sense. What would the "span" of $A$'s columns / rows even mean?
I also have no idea what the nullspace of $A$ or $A^T$ would represent - if it represents anything at all - nor what
$u_{r+1},..,u_{m}$ being an orthonormal basis for $ker(A)$
or $v_{r+1},..,v_{n}$ being an orthonormal basis for $ker(A^T)$
implies - again, if anything at all.


texts:
This is more like a general NLP question.
What is the appropriate input to train a word embedding namely Word2Vec? Should all sentences belonging to an article be a separate document in a corpus? Or should each article be a document in said corpus?
This is just an example using python and gensim.

Corpus split by sentence:

SentenceCorpus = [["first", "sentence", "of", "the", "first", "article."],
                  ["second", "sentence", "of", "the", "first", "article."],
                  ["first", "sentence", "of", "the", "second", "article."],
                  ["second", "sentence", "of", "the", "second", "article."]]

Corpus split by article:

ArticleCorpus = [["first", "sentence", "of", "the", "first", "article.",
                  "second", "sentence", "of", "the", "first", "article."],
                 ["first", "sentence", "of", "the", "second", "article.",
                  "second", "sentence", "of", "the", "second", "article."]]

Training Word2Vec in Python:

from gensim.models import Word2Vec

wikiWord2Vec = Word2Vec(ArticleCorpus)


texts: Can you use two different datasets as train and test sets with countVectorizer and test_train_split?

texts:
So I managed to run my code on a combination of train data and validation data, but now I need to create a text file that contains the predictions for the test data and I just don't understand how. Is there any way to make the X_train work with train_data and X_test with test_data? I think that would solve my problem but I can't find how or if it is even possible.

train_data = np.genfromtxt('train_samples.txt', delimiter = '\t', dtype = None, encoding = 'utf-8', names = ('id', 'text'),
                               comments = None)

    train_labels = np.genfromtxt('train_labels.txt', delimiter='\t', dtype = None, names = ('id', 'label'))

    test_data = np.genfromtxt('test_samples.txt', delimiter = '\t', dtype = None, encoding = 'utf-8', names = ('id', 'text'),
                              comments = None)

    validation_data = np.genfromtxt('validation_samples.txt', delimiter='\t', dtype = None, encoding='utf-8',
                                    names = ('id', 'text'), comments = None)
    validation_labels = np.genfromtxt('validation_labels.txt', delimiter = '\t', dtype = None, names = ('id', 'label'))

    for x in range(len(train_data)):
        train_data[x][0] = train_labels[x][1]

    for x in range(len(validation_data)):
        validation_data[x][0] = validation_labels[x][1]

    train_data_text = np.append(train_data['text'], validation_data['text'])
    train_data_labels = np.append(train_data['id'], validation_data['id'])

    # show shape of training data
    cv = CountVectorizer()
    word_count_vector = cv.fit_transform(train_data_text)
    print(word_count_vector.shape)

    # train_data = np.concatenate((train_data, validation_data))
    X = cv.fit_transform(train_data_text).toarray()
    y = pd.get_dummies(train_data_labels)
    y = y.iloc[:, 1].values

    # Train Test Split
    from sklearn.model_selection import train_test_split

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=0)

    # Training model using Naive bayes classifier
    from sklearn.naive_bayes import MultinomialNB

    results = MultinomialNB().fit(X_train, y_train)

    y_pred = results.predict(X_test)
    print(y_pred)

    from sklearn.metrics import accuracy_score

    # Evaluate accuracy
    print(accuracy_score(y_test, y_pred))

    classifier.fit(X_train, y_train)
    y_pred = classifier.predict(X_test)


texts:
We are trying to figure out what is the best approach for us to train a ML model to identify Authors. We have structured metadata of authors (given name, surname. etc) and the task is to train a model which recognizes the author when we receive a new name. The challenge is to identify names written in different ways e.g. John Smith, J Smith, J.Smith, Jon Smith etc.
We have nearly 130000 records as input data set.


texts: Training Dataset for Sentiment Analysis of Movie Reviews

texts:
I am currently working on sentiment analysis using Python. I wanted to find whether reviews given for a movie is positive or negative based on sentiment analysis. I have found a training dataset as provided in this link.
This dataset have reviews and a score with 1 indicating review is positive and 0 indicating movie review is negative, but it has less number of records. I have a test dataset which I will predicting based on training set. My test dataset has complex and long words for which my python ML model sometimes gives positive result for a negative reviews (returning result as 1 for negative review). I am looking for better dataset to train my model, so that my model can predict well. Could you please advise me any good/large and effective training dataset to use for this scenario? If you could share any links that would be great. The training data can be in format as 1 for positive reviews and 0 for negative reviews or even polarity like pos for positive reviews or neg for negative reviews.


texts: Improving a CNNs accuracy - help & advice

texts:
So I have created a CNN for image classification, and I train and test it with two datasets. One contains 9,339 images and the other 9,100 images. The first model which I designed gave an accuracy of 94% on the first dataset (with 9,339 images) and 86% on the other (with 9,100 images).
This is the code for the model:
detect_model.add(Conv2D(16, kernel_size=(3, 3), activation='relu', input_shape=(64,64,3)))
detect_model.add(MaxPooling2D(pool_size=(2, 2)))
detect_model.add(Conv2D(32, (3, 3), activation='relu'))
detect_model.add(MaxPooling2D(pool_size=(2, 2)))
detect_model.add(Dropout(0.25))
detect_model.add(Flatten())
detect_model.add(Dense(128, activation='relu'))
detect_model.add(Dropout(0.25))
detect_model.add(Dropout(0.5))
detect_model.add(Dense(50, activation='relu'))
detect_model.add(Dense(num_classes, activation='softmax'))
detect_model.compile(loss='categorical_crossentropy', optimizer = 'adam', metrics=['accuracy'])

I was very impressed with the results on the first dataset in particularly. I am trying to improve the accuracy now and have tried many things, but none of them have improved the accuracy:

Adding BatchNormalization layers after the activation layers

BatchNormalization doesn't help. It trains the model much faster but gives me a high training accuracy but low validation accuracy after every epoch ...

Changing the epoch from 10 to 15

The model slows down in training and validation accuracy scores after about the seventh or eighth epoch and still produces the same results on the second dataset (I haven't tried doing this on the first dataset)

Changing the filters from 16 to 32 and higher in subsequent conv2d layers.

The model still produces the same accuracy on the first and second dataset.

Changing drop out rates

Still no change

Adding more depth (add more conv2d and max-pooling layers with more filters, still don't seem to like this)

Has anyone got any suggestions for me? Any clear and helpful information will definitely be upvoted and, of course, appreciated.


texts:
I am attempting to implement YOLO v3 in Tensorflow-Keras from scratch, with the aim of training my own model on a custom dataset. By that, I mean without using pretrained weights. I have gone through all three papers for YOLOv1, YOLOv2(YOLO9000) and YOLOv3, and find that although Darknet53 is used as a feature extractor for YOLOv3, I am unable to point out the complete architecture which extends after that - the "detection" layers talked about here. After a lot of reading on blog posts from Medium, kdnuggets and other similar sites, I ended up with a few significant questions:

Have I have missed the complete architecture of the detection layers (that extend after Darknet53 used for feature extraction) in YOLOv3 paper somewhere?
The author seems to use different image sizes at different stages of training. Does the network automatically do this upscaling/downscaling of images?
For preprocessing the images, is it really just enough to resize them and then normalize it (dividing by 255)?

Please be kind enough to point me in the right direction. I appreciate the help!


texts: Fine tuning CNN with imbalance data gives good results - not sure why

texts: I'm fine tuning with caffenet for a 3 classes classification problem. I have 100 instances of class A, 90 instances of class B and 30 instances of class C. I thought that my net would be biased toward classes A and B but I'm actually getting quite good results. I know that caffenet doesn't take care of imbalance data for me. Maybe it has to do with the fact that my entire train set fits into one batch? Or maybe it has to do with the fact that I'm not really training from scratch but mostly using caffenet's already learned weights? Thanks


texts: Doc2vec most similar document to a query string

texts:
I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title and abstract. Now, I want to do a search on all of the professors and find which professor is the most similar to a query string, such as "deep learning" or "computer networking". I have tried to use the infer_vector() to create a doc2vec representation of the query string using the already generated model and calculate the cosine similarity between the vectors. But I got terrible results. For example, when I search for "computer networking", it will give me the result of professor from History.
Is there any recommendation of how to find most similar document to a query string?


texts: Shall I use the Euclidean Distance or the Cosine Similarity to compute the semantic similarity of two words?

texts:
I want to compute the semantic similarity of two words using their vector representations (obtained using e.g. word2vec, GloVe, etc.). Shall I use the Euclidean Distance or the Cosine Similarity?

The GloVe website mentions both measures without telling the pros and cons of each:

  The Euclidean distance (or cosine similarity) between two word vectors provides an effective method for measuring the linguistic or semantic similarity of the corresponding words.


texts: geo spatial clustering based on another feature

texts:
I have data about houses for sale, that I present over a map.

Each house has coordinates ([lat,lng]) and other features. The data is only for one country, so no need to address the 180deg world wrap.

I want to cluster neighbourhoods, out of those houses, and base the the neighbourhood borders on prices (sqm price).

It's really easy to see those clusters when I paint the houses with gradient color - from cheap to expensive, but I can't find a trivial way to do it mathematically.

Basically, what I want is price clustering, but with a limitation on the clustering algorithm so there won't be any geo-location overlap.

Another way to look at it - I want to blur the data points and create low-resolution areas.

Not sure where to go with this, any help will be appreciated.


texts: Retrieving date ranges with the r platform

texts:
Using r if I have a data frame that contains dates and temperatures.

I would like to pull out all temperatures for January how would I go about

phrasing this.  I've tried may ways such as

louisville_weather <- filter(louisville_weather$DATE <= 2013-01-31)


texts: How to solve XNOR problem with neural network?

texts:
How do we design a neural network with one hidden layer, two hidden neurons and an output neuron that implements an XNOR function? The truth table of XNOR is given below:

And how to provide weight and bias coefficients of this network?


texts: Creating obligatory combinations of variables for drawing by random forest

texts:
Problem

For my machine learning task, I create a set of predictors.
Predictors come in "bundles" - multi-dimensional measurements (3 or 4 - dimensional in my case).

The hole "bundle" makes sense only if it has been measured, and taken all together.

The problem is, different 'bundles' of predictors can be measured only for small part of the sample, and those parts don't necessary intersect for different 'bundles'.

As parts are small, imputing leads to considerable decrease in accuracy(catastrophical to be more accurate)

Possible solutions

I could create dummy variables that would mark whether the measurement has taken place for each variable. The problem is, when random forests draws random variables, it does so individually.

So there are two basic ways to solve this problem:
1) Combine each "bundle" into one predictor. That is possible, but it seems information will be lost.
2) Make random forest draw variables not individually, but by obligatory "bundles".

Problem for random forest

As random forest draws variables randomly, it takes features that are useless (or much less useful) without other from their "bundle". I have a feeling that leads to a loss of accuracy.

Example

For example I have variables a,a_measure, b,b_measure.
The problem is, variables a_measure make sense only if variable a is present, same for b. So I either have to combine aand a_measure into one variable, or make random forest draw both, in case at least one of them is drawn.

Question

What are the best practice solutions for problems when different sets of predictors are measured for small parts of overall population, and these sets of predictors come in obligatory "bundles"?

Thank you!


texts: Text classification and clustering with complete date imbalance

texts:
I have a set of scientific papers of authors who have common research interests from PUBMED and I would like to:

Clustering papers and extracting features from them in order to find other authors with similar research interests in PUBMED
Develop a classification algorithm to determine if a given author
has the similar research interest to those in my original set.
Really, I am trying find new authors and be able to tell if an
author that I discovered, is similar to my original set of authors.

Unfortunately because my data is only one class, I have an extreme data imbalance problem.  How should I fix this?

Thus as far as I have looked at the frequency of terms in the papers with mixed results.  I have also been using unsupervised learning to extract features with mixed results.

My set of papers and authors are assumed to share similar research interests. Do I need papers and authors that I know aren’t in the set in order to build a classification algorithm?

Should I look at a paper as a whole or should I look at the papers written by each author for classification, instead of looking at the papers as a group?

I suppose I could randomly pick other papers and use those as the examples of other fields. Of course, picking paper at random there is the chance of false negatives.

I am using R with the tm package to do the mining.


texts:
I'm taking the abs of all elements, compute the mean, subtract it off from the original values. I just feel that this is not correct and can change the vectors.
I'm also dividing by the standard deviation, but I'm quite confident about this, knowing that this is pure rescaling of complex values.

Any guidelines on how to do this?

PS: My concern eminates from thinking of each complex valued element as a vector in 2D plane, and this subtraction could change where it is pointing to.


texts: How to test accuracy of an unsupervised clustering model output?

texts:
I am trying to test how well my unsupervised K-Means clustering properly clusters my data.  I have an unsupervised K-Means clustering model output (as shown in the first photo below) and then I clustered my data using the actual classifications.

The photo below are the actual classifications.  I am trying to test, in Python, how well my K-Means classification (above) did against the actual classification.

For my K-Means code, I am using a simple model, as follows:

kmeans = KMeans(n_clusters=4, random_state=0).fit(myData)
labels = kmeans.labels_

What would be the best way for me to compare how well my unsupervised KMeans clustering model did against the actual classifications?


texts: Experiments on time spent by the visual cortex

texts:
I'm working on a object detection problem, using convolutional neural networks which have a lot in common with how the human visual cortex works. I've previously seen remarks on tasks in computer vision like "if a human can do it in x milliseconds, it can easily be solved by a computer".

I was thinking of setting up an experiment where I flash an image for a set amount of time and the subject records whether (s)he detected the object or not. I would then have multiple subjects, a large set of images, and a varying flash length. Thus I could kind of quantify how hard my computer vision task is.

My question is whether you know of any research, papers, books or other resources on this subject? Any resources specifically linked to neural networks and deep learning would be especially helpful.


texts: SRCNN - how to get a color image output training only with luminance

texts:
I'm quite new to convolutional neural network, applied to super-resolution. I read this review article, itself based on this paper as an attempt to understand it better.

In the review, the author says :

  where c is [the] number of channels of the image [...].
  In this case, c=1 [...]

But the example shows a color image. So, first, I asked myself "How a color image could have only 1 channel ?".

Then I read the following in the paper :

  The majority of SR algorithms [...] focus on gray-scale or single-channel
  image super-resolution. For color images, the aforementioned
  methods first transform the problem to a different
  color space (YCbCr or YUV), and SR is applied
  only on the luminance channel.

Ok, so the network only works with luminance. Therefore how can it generate a color image as an output ? Converting a full color system to only luminance, a part of the information on the color is lost. How does the network get it back in the process ?